Coordinate system registration system and method based on 2D X-ray imaging device and augmented reality device

Abstract

A system and method for assisting musculoskeletal surgery is provided. Generally, a first X-ray image showing a first object is received, the first X-ray image being generated by an imaging device. Based on a three-dimensional representation of the first object, an imaging orientation of the first X-ray image is determined, and the imaging orientation of the first X-ray image is associated with a coordinate system of the first X-ray image. Information regarding the orientation of the imaging device when the first X-ray image was generated is then received. This information includes geometric aspects of the imaging device and is provided by an augmented reality device associated with a global coordinate system. Furthermore, based on registration of the coordinate system of the first X-ray image with the global coordinate system, a position and orientation of the first object relative to the global coordinate system are determined.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION The present invention relates to the field of systems and methods in musculoskeletal therapy. In particular, the present invention relates to computerized evaluation of X-ray images and visualization of information within a physician's field of view to assist the physician in performing musculoskeletal therapy. The method according to the present invention may be implemented as a software program product executable on a processing unit of the system. [Background technology]

[0002] When a bone is fractured, the bone fragments may be stabilized with an implant, such as an intramedullary nail inserted into the bone's medullary canal or a bone plate fixed to the bone's surface, to aid in healing. The surgical procedure to implant such an implant is minimally invasive and may require repeated x-ray imaging to ensure the surgeon correctly positions the implant. The implant may also be connected to one or more sub-implants, such as screws or blades.

[0003] During a surgical procedure, it may be necessary to determine the location, shape, and dimensions of an object (e.g., a surgical instrument, an implant, or a bone structure) shown in an x-ray image. For example, a surgeon may need to determine the length of an implant blade by measuring an anatomical structure shown in an x-ray image. However, reliable dimensional measurements based on x-ray images require calibration.

[0004] For example, it has been proposed to utilize fiducials attached to X-ray imaging devices to calibrate acquired X-ray images. Furthermore, the art has proposed the use of at least one additional fiducial attached to instruments. Such fiducials can assist in determining the three-dimensional position of implants and surgical instruments, thereby enabling computer-assisted surgery systems to determine and display the relative position of the implant and bone. Furthermore, fiducials may be required to align images of the femoral head acquired from different directions to provide a three-dimensional representation of the femoral head.

[0005] Sufficiently accurate reduction of a fracture is essential for achieving a satisfactory clinical outcome in any osteosynthesis surgery. Satisfactory healing usually occurs only if the reduction is performed correctly. Reduction of long bones, in particular, is often difficult to assess intraoperatively, particularly with regard to the accuracy of the anteversion angle. Incorrect anteversion is often only noticed after the surgery is completed and the patient is able to stand again. Even if the fracture itself has healed adequately at this stage, incorrect anteversion can cause significant discomfort to the patient. Therefore, adequate accuracy of anteversion is essential for achieving a satisfactory clinical outcome, especially in osteosynthesis of the femur and tibia. In fact, incorrect anteversion is one of the leading causes of revision surgery.

[0006] Prior art techniques have proposed different methods for determining anteversion. In the case of femoral and cephalomedular nails, one method involves manually checking whether the knee is pointing toward the operating room ceiling and subjectively determining whether the nail axis and the screw that should intersect with the center of the femoral head form an angle of approximately 10 degrees with the operating room floor. Another method, proposed in U.S. Patent Application Publication No. 2015 / 0265361 A1, uses two fiducials—metal markers on the distal and proximal portions of the femur—and two proximal and one distal x-ray images showing each fiducial.

[0007] The main difficulty in determining anteversion is that long bones are too long to fit on a single radiograph. Furthermore, the geometries required to determine anteversion are located at the most proximal and most distal portions of the bone—for example, the femur's neck and condyles. Therefore, the geometries shown separately on the proximal and distal radiographs must be correlated.

[0008] The relationship between separate X-ray images can be obtained based on knowledge of the imaging direction of each image. WO 2020 / 038917 A1 describes a concept for determining an imaging direction in which specific anatomical features can be identified by an algorithm in an X-ray image, and based on the relative positions of these features in the image, it can be determined whether the actual imaging direction matches the intended imaging direction. For example, in insertion point assistance for inserting a nail into the bone marrow, an oblique lateral view may be required to evaluate the current position of an opening instrument relative to the detected bone axis. In the acquired X-ray image, for example, the position of the greater trochanter and the position of the distraction bone axis can be compared to determine whether the imaging direction is appropriate. If these positions are close, the imaging direction can be deemed acceptable by the algorithm. If these positions are not close, the imaging direction is not appropriate. Summary of the Invention

[0009] It is an object of the present invention to provide a system and method that improves a physician's ability to assess the relative position and orientation of two objects in a treatment area. It is also an object of the present invention to provide a system and method that provides information about the spatial position and orientation of objects in an X-ray image relative to the three-dimensional space of the operating room. Based on this information, it is possible to provide information about the relative position of two objects.

[0010] Solutions to these objects are provided by the subject matter of each independent claim. Some embodiments are set out in the respective dependent claims.

[0011] Generally, the system includes a processing device, and the method is implemented by a computer program product, i.e., computer software. The computer program product executable on the processing device includes a set of instructions for receiving a first X-ray image, generated by an imaging device, showing a first object. The first object may be an implant or a tool. Furthermore, the first object may be a bone. In particular, the first object may be a bone fragment. It should be understood that the first object may only be partially visible in the X-ray image. The imaging device may be a C-arm-based X-ray imaging device.

[0012] The computer program product further includes a set of instructions for determining an imaging orientation of a first X-ray image based on a three-dimensional representation of the first object, the imaging orientation of the first X-ray image being associated with the coordinate system of the first X-ray image. Determining the imaging orientation of the X-ray image may be understood, for example, as removing three degrees of freedom of the object depicted in the X-ray image, i.e., determining the rotation of the object in the coordinate system of the X-ray image. In the context of the present disclosure, the three-dimensional representation is a model of the object, which may be determined in advance or generated during a treatment procedure. The model may be a three-dimensional dataset of the implant or tool to be used. In the case of bone, the model may be a statistical model, such as a statistical shape model or a statistical appearance model. The bone model may be a three-dimensional representation of the bone used for bone detection and / or pose estimation. The three-dimensional representation may be the result of a preoperative scan, such as a computed tomography (CT) scan, or a cone-beam-based three-dimensional image, which may be acquired intraoperatively, and may be an approximate representation of the bone with or without prior knowledge. The computer program product may further include a set of instructions for generating at least one of a three-dimensional representation of the first object or a three-dimensional representation of the second object. In other words, a "three-dimensional representation" may refer to a complete or partial description of a three-dimensional volume or surface, and may also refer to selected geometric aspects, such as an axis, a plane, a two-dimensional manifold, a sphere, etc. While it is possible to determine complete three-dimensional information, including scaling, about the three-dimensional surface or volume of an object, in many applications it is sufficient to determine only selected geometric aspects or ratios of those aspects (e.g., width to length ratios).

[0013] The computer program product includes a set of instructions for receiving information regarding the orientation of the imaging device at which the first X-ray image was generated. This information, which may be partial information and includes geometric aspects of the imaging device, is provided by an augmented reality device associated with a global coordinate system. In this case, the "orientation" of the imaging device is intended to mean the spatial position and orientation of the imaging device within the operating room space. For example, the orientation may include, for example, the distance from the imaging device to the floor and walls of the operating room, and the angles the radiation source and radiation detector make with respect to the floor and walls of the operating room. In the context of the present disclosure, the global coordinate system relates to the operating room, i.e., the room in which the imaging device is located. The floor and walls of the operating room and the devices within the room can be detected by sensors in the augmented reality device, such as a head-mounted display, so that the augmented reality device can provide information that enables the computer program product in the system to define the global coordinate system. For example, the head-mounted display can detect the operating room, which defines the global coordinate system, and can further detect the distance and angle of the X-ray radiation detector of the imaging device relative to the global coordinate system.

[0014] The pose information may include six translation parameters and six rotation parameters. In particular, the pose information may consist of fewer than six parameters.

[0015] The computer program product further includes a set of instructions for determining a position and orientation of the first object relative to the global coordinate system based on registration of the coordinate system of the first x-ray image and the global coordinate system.

[0016] When both the first object and the tool are displayed in the X-ray image, the computer program product may further include a set of instructions for removing at least two, e.g., three, four, or five, degrees of freedom of the position and orientation of the first object relative to the tool, rather than removing all degrees of freedom of the position and orientation of the tool in the global coordinate system, and a set of instructions for identifying a geometric aspect in the coordinate system of the first X-ray image. In three-dimensional space, an object such as a tool has six degrees of freedom, three of which are translational degrees of freedom and three of which are rotational degrees of freedom. The spatial position and orientation of the object can be determined by removing the degrees of freedom, i.e., by defining translation values ​​along three coordinate axes and rotation values ​​around three axes. In this case, ambiguity is tolerated because only four or two of the possible six degrees of freedom are removed. The geometric aspect in the coordinate system of the first X-ray image may be a two-dimensional manifold, a curve, a point, or a set of points. Despite the above ambiguities, determining the position and orientation of the first object relative to the global coordinate system may be performed based on the relationship between geometric aspects of the imaging device and geometric aspects in the coordinate system of the x-ray image.

[0017] The computer program product may further include a set of instructions for receiving a second X-ray image of a second object, the second X-ray image being generated by an imaging device, and determining an imaging orientation for the second X-ray image based on the three-dimensional representation of the second object, the imaging orientation for the second X-ray image being associated with a coordinate system for the second X-ray image. The above description regarding the first object in the first X-ray image also applies to the second object in the second image. Similarly, the computer program product may include a set of instructions for receiving information regarding the orientation of the imaging device when the second X-ray image was generated, the information including geometric aspects of the imaging device, provided by an augmented reality device associated with a global coordinate system. When the second X-ray image is provided, the position and orientation of the second object relative to the first object in the global coordinate system may be determined based on registration of the coordinate system of the second X-ray image with the global coordinate system.

[0018] As described above, the first object may be an implant, a tool, a bone, or a fragment or portion of a bone, and the first object may only be partially visible in the X-ray image. The second object may also be one of these objects (e.g., whether another portion of the same bone is fractured), and multiple different combinations of first and second objects may occur. For example, the first object may be a bone and the second object may be an implant. Alternatively, the first object may be an implant and the second object may be a tool such as a drill or a bone screw. Furthermore, the first object and the second object may both be a fragment or portion of a single bone. In either case, information about the spatial position and orientation of the two objects relative to each other may be useful.

[0019] If the first object is a first fragment of a fractured bone and the second object is a second fragment of the fractured bone, the computer program product may further include a set of instructions for determining at least one aspect of the group consisting of: bone length, anteversion angle, rotation angle, centrum-collum-diaphyseal (CCD) angle, length-to-width ratio of the bone or portion thereof, and curvature of the bone.

[0020] The object's position and orientation information may be provided to the physician on a display screen. For example, the display may be positioned near the treatment site, allowing the physician to view the treatment site and the display only by slightly changing their line of sight. The information may be projected onto an X-ray device, giving the physician the same visual impression as an X-ray imaging device. Alternatively, an augmented reality device, such as a head-mounted display, may be used, where the head-mounted display is semi-transparent, allowing the physician to view both the treatment site and the information displayed on the display. The computer program product further includes a set of instructions for providing a visualization on the augmented reality device, the visualization being at least one of the group consisting of a three-dimensional representation of a first object, a three-dimensional representation of a second object, and a tool, the visualization being displayed at a position and orientation determined in a global coordinate system. It will be appreciated that the augmented reality device may also display implants or tools already inserted into the patient. It will also be appreciated that other objects, such as tools, implants, or bones, may be visualized on the augmented reality device, and that other objects may not be depicted in the X-ray image but may only be identified by the augmented reality device in the global coordinate system.

[0021] Additionally, the computer program product may include a set of instructions for determining an insertion path for a tool or implant and providing a visualization of the insertion path on an augmented reality device. The insertion path is displayed at a position and orientation determined in a global coordinate system. For example, the insertion path may be visualized as a volume, a curve, or a point on the display of the augmented reality device. By determining such a path in a global coordinate system, the path can be virtually maintained in a three-dimensional position and orientation even as the augmented reality device is moved. When such a path is present in a physician's field of view, the physician can easily manipulate a tool, such as a scalpel, drill, or implant, to follow the path as it is inserted into a patient.

[0022] Alternatively or additionally, the computer program product may further include an instruction set for receiving information regarding a predetermined position and orientation of at least one of the first object or the second object and providing a visualization of the information on an augmented reality device. For example, the anatomically correct position and orientation may be visualized in addition to the actual three-dimensional position and orientation of the two fragments. The correct position and orientation may be visualized as an outline of the object. Such visualization facilitates repositioning of the fragments. A guide to the correct position may be displayed, for example, by an augmented reality device using arrows and / or values ​​at appropriate positions.

[0023] Additionally, the computer program product may include a set of instructions for identifying the current state of bone healing and providing information suitable for guiding the next steps in bone healing. For example, the system may recognize by interpreting an X-ray image that an implant has already been inserted into the patient. Following a predetermined procedure, the system may display (or sound) the next step to be performed.

[0024] Furthermore, the doctor and / or the system (via the augmented reality device) may detect that the patient or at least a part of the patient during treatment has moved. In that case, it may first be determined whether the movement will affect the treatment. For example, the patient's leg may move in the direction of the drilling due to pressure from the drilling (e.g., pressure on the bone). It can be assumed that such movement, if any, will have little effect on the treatment, so the doctor may continue drilling. As another example, the bone may tilt while drilling a screw hole. Such tilt may require an adjustment of the drilling trajectory. Depending on the actual case, the doctor or the system may suggest generating a new X-ray image. For example, based on information received from the augmented reality device, unintended movement may be detected. Based on the new X-ray image, the system may ultimately provide guidance regarding further progress of the treatment.

[0025] The computer program product as described above may be executed on a processor of the system. In addition to the processor, the system may include an augmented reality device and / or an imaging device. In particular, the system may include a head-mounted display and / or a C-arm-based X-ray imaging device. It is also envisioned that the imaging device may include multiple separate imaging devices. Alternatively, the imaging device may include multiple combinations of radiation sources and radiation detectors.

[0026] It should be noted that the processing device may be implemented by a single processor that performs all steps of the process, or by a group or groups of processors that need not be co-located. For example, with cloud computing, the processors may be located anywhere. For example, the processing device may be divided into (i) a first sub-processor that processes X-ray image data (including determining the imaging direction based on the X-ray image), (ii) a second sub-processor that determines a global coordinate system based on information received from an augmented reality device, and (iii) a further processor that controls a monitor that visualizes the results or a speaker that provides acoustic instructions to the user. Either of these processors or the further processor may also control, for example, the operation of the C-arm of the X-ray imaging device.

[0027] In some embodiments, the apparatus may further comprise storage means, for example providing a database for storing X-ray images, it being understood that such storage means may also be provided over a network to which the system can be connected, and that data relating to the neural net is received over said network.

[0028] The device may further comprise input means for manually determining or selecting the location or portion of an object, such as the outline of a bone in an X-ray image, for example, for measuring distances in the image. Such input means may include, for example, a computer keyboard, a computer mouse or touch screen for controlling a pointing device such as a cursor on a monitor screen, a controller, a hand or eye tracking device for a head-mounted display, etc., which may also be incorporated into the device.

[0029] The computer program product is preferably loaded into the random access memory of a data processor. Thus, the data processor or processing unit of the system according to the embodiment is capable of performing at least part of the above-described process. Furthermore, the present invention relates to a computer-readable medium, such as a CD-ROM, for storing the disclosed computer program. However, the computer program may be presented via a network, such as the World Wide Web, and downloaded from such a network into the random access memory of the data processor. Furthermore, the computer program may be executed on a cloud-based processor, and the results may be presented via a network.

[0030] As is clear from the above description, a main aspect of the present invention is the processing of X-ray image data and coordinate systems. The method described herein should be understood as a method for assisting in the surgical treatment of a patient. Thus, the method may, depending on the embodiment, not include any steps involving surgical treatment on an animal or human body.

[0031] It should be noted that the embodiments are described based on different subject matters. In particular, some embodiments are described based on method-type claims (computer programs), and other embodiments are described based on apparatus-type claims (systems / apparatuses). However, those skilled in the art will understand from the above and following description that, unless otherwise specified, any combination of features belonging to one subject matter and any combination between features belonging to multiple different subject matters are disclosed in the present application. [Brief explanation of the drawings]

[0032] [Figure 1] 1 shows a flowchart of a method implemented by a computer program product. [Figure 2] 1 shows a schematic diagram of a system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0033] The above-defined aspects and further aspects, features and advantages of the present invention can also be derived from the embodiments / examples described below, but the present invention is not limited thereto. It will be understood that the aspects in the following embodiments / examples may be combined to obtain advantages. That is, not all aspects are described in detail in the following embodiments / examples.

[0034] Example 1.1 An imaging device acquires an anterior-posterior x-ray image, and an augmented reality device detects a first point on the imaging device in a global coordinate system. Using a model of the bone, the system detects the bone in the x-ray image in the image coordinate system, and a second point in the image coordinate system is identified that is known to correspond to the first point in the global coordinate system. The system receives information about the first point from the augmented reality device and, based on given knowledge of how the detected bone must be oriented in the OR to estimate how the image coordinate system is rotated relative to the global coordinate system, determines that the anterior-posterior image was created and determines the location and orientation of the implant in the global coordinate system.

[0035] A first point in the global coordinate system is placed on a second point in the image coordinate system, thereby eliminating three translational degrees of freedom.

[0036] This embodiment can be considered as a method including at least steps S1 to S4 of FIG.

[0037] Example 2.1 When placing the femoral nail and drill, identify the natural axis of the drill.

[0038] When the system detects the orientation of the nail in the first image coordinate system of the imaging device that acquires the first X-ray image, it detects the intrinsic axes of the drill in the first image coordinate system (i.e., removes four degrees of freedom). The augmented reality device detects the intrinsic axes of the drill in the global coordinate system (i.e., removes four degrees of freedom) and detects a first geometric aspect of the imaging device in the global coordinate system. Since it is known that the first geometric aspect corresponds to a second geometric aspect in the first image coordinate system, the system can register the first image coordinate system and the global coordinate system by removing the four degrees of freedom using the detection results of the intrinsic axes of the drill in both coordinate systems and further removing the remaining two degrees of freedom, consisting of rotation about the intrinsic axes and translation on the intrinsic axes, using the correspondence between the first geometric aspect and the second geometric aspect. As a result, the system determines the transformation between the first image coordinate system and the global coordinate system.

[0039] This embodiment may be viewed as being reflected in the method of FIG. 1, particularly including steps S5 through S7.

[0040] Given the position and orientation of the drill in the global coordinate system at the time of the X-ray image acquisition, the augmented reality device may update an estimate of the drill's intrinsic axis based on detecting and / or tracking geometric aspects of the drill and / or power tool. This information can be used to estimate the trajectory of the drill relative to the nail or the trajectory of the drill relative to the bone into which the nail is inserted.

[0041] Given an estimate of the drill's position and orientation in the global coordinate system, the system can guide the surgeon in adjusting the drill's rotation or position. It can also provide feedback, such as showing the deviation of the drill's current position or rotation from a target position or rotation. This can include feedback after drilling has begun. This can be used with or without a new x-ray image, which can be used to recalibrate the estimated drill position or rotation.

[0042] Figure 2 shows an imaging device (ID) acquiring an X-ray image of a nail (N) inserted into bone. A drill (D) can be positioned on the bone with its inherent axis (IA). The drill (D) and a portion of the nail (N) are visible in the X-ray image.

[0043] The Augmented Reality Device (ARD), in this case a Head Mounted Display (HMD), observes the placement with a Camera (C) that tracks it in a Global Coordinate System (GCS) fixed in the operating room. The ARD detects the drill machine (DM) to detect the intrinsic axis (IA) of the drill, and detects a point on the image detector of the Imaging Device (ID) as an example of the first geometric aspect in the Global Coordinate System (GCS).

[0044] The system processes the X-ray image in an image coordinate system (ICS) that is fixed relative to the X-ray machine (IC) at the time the image was recorded. The system detects the position and orientation of the nail and drill in the image coordinate system. Furthermore, the system identifies points on the image plane as examples of the second geometric aspect in the image coordinate system (GA2_ICS).

[0045] Second, we take advantage of the fact that the intrinsic axis (IA) of the drill is detected in both coordinate systems. In this case, the relationship between the first geometric aspect and the second geometric aspect is known and they are the same point. By registering the image coordinate system (ICS) and the global coordinate system (GCS), the system can detect the position and orientation of the nail and drill in the global coordinate system.

[0046] Example 2.2 For placement of an intraosseous femoral nail and proximal targeting device implanted in the femur.

[0047] In an imaging device that acquires X-ray images, the system detects a first partially incorrect orientation of the nail in an image coordinate system and a first partially incorrect orientation of the target device. The augmented reality device detects the orientation of the target device in a global coordinate system and a first geometric aspect of the imaging device. A second geometric aspect is identified in the image coordinate system, and the system detects a second orientation of the target device in the image coordinate system using the relationship between the first geometric aspect and the second geometric aspect, thereby reducing the estimation error of the first partially incorrect orientation of the target device. The system estimates a second orientation of the nail using the spatial relationship between the nail and the target device, thereby reducing the estimation error of the first partially incorrect orientation of the nail. Furthermore, the system estimates the orientation of the nail in the global coordinate system using the spatial relationship between the target device and the nail.

[0048] Using estimates of the nail and targeting device pose in the global and image coordinate systems, the system can guide the surgeon in adjusting the nail position.

[0049] Additionally, the augmented reality device can estimate the pose of the target device after the x-ray image is acquired, allowing the system to identify movement of the target device. This updated estimate of the pose of the target device in the global coordinate system can be used to guide the surgeon in positioning the target device or nail.

[0050] The positioning of the nail is guided in relation to a pre-estimated anatomy (eg, the proximal femur), which suggests to the surgeon the location or orientation of the nail within the bone.

[0051] The proposed nail position within the femur may be displayed on the augmented reality device by visualizing the current position or orientation of the nail in a global coordinate system or relative to the bone, which may include the blade to be inserted through the nail after placement is complete, or the distance between the bone surface and the trajectory based on an estimate of the current nail position or orientation.

[0052] Example 2.3 When placing the femoral nail and drill, identify the natural axis of the drill.

[0053] In the imaging device that acquires the first X-ray image, the system detects the nail's orientation in the first image coordinate system and detects a two-dimensional projection (i.e., two degrees of freedom removed) of the drill's intrinsic axis in the image. This projection of the intrinsic axis is a straight line in the first X-ray image, which directly corresponds to a three-dimensional plane P in the first image coordinate system. The augmented reality device detects the drill's intrinsic axis in the global coordinate system (i.e., four degrees of freedom removed). If a first set of two points on the imaging device (e.g., the center of the image intensifier and the center of the X-ray source) detected by the augmented reality device in the global coordinate system corresponds to a second set of two points in the first image coordinate system, the system uses the relationship between these two sets of points to register the first image coordinate system with the global coordinate system. Here, any remaining ambiguity in the registration is resolved by the condition that the drill's intrinsic axis must lie on plane P of the first image coordinate system in the global coordinate system. Note that the diameter of a drill may be very small. Instead of a drill, a needle or similar thin tool may be sufficient for registering the two coordinate systems.

[0054] Types of geometric aspects and their relationships

[0055] Examples of the first geometric aspect include a point on the image intensifier in the global coordinate system, a point on the X-ray source in the global coordinate system, an axis passing through the center of the X-ray source and the center of the image intensifier in the global coordinate system, a plane in the global coordinate system in which that axis lies, and a plane within the image intensifier that is parallel to the image in the global coordinate system.

[0056] Examples of the second geometric aspect include a point on the image intensifier in the first image coordinate system, a point on the X-ray source in the first image coordinate system, an axis passing through the center of the X-ray source and the center of the image intensifier in the first image coordinate system, a plane in which the axis lies in the first image coordinate system, and a plane within the image intensifier that is parallel to the image in the first image coordinate system.

[0057] The relationship between the first and second geometric aspects is used to remove ambiguity or degrees of freedom to achieve registration. These relationships include (but are not limited to) the following options: The first geometric aspect is a set of one or more points, and the second geometric aspect is also a set of one or more points. Each point on the first geometric aspect must be at the same location as a point on the other geometric aspect. More specific examples of these geometric aspects are as follows: Two points on the first geometric aspect, the center of the image plane in the global coordinate system and the center of the X-ray source of the imaging device, must correspond to two points on the second geometric aspect in the first image coordinate system after registration. Any two points on the geometric aspects may be known exactly or approximately, in which case registration can be approximated by minimizing the distance between the geometric aspects. The distance is the sum of the L2 norms of the differences between corresponding points. The first geometric aspect is an axis in the global coordinate system. The second geometric aspect is an axis in the first image coordinate system. The relationship between these geometric aspects is such that after registration is complete, the two axes must be the same, and the first axis must lie on the second axis. An example of these axes is the central X-ray beam of the imaging device in the global coordinate system, which is the first geometric aspect, and the central X-ray beam of the imaging device in the first image coordinate system, which is the second geometric aspect. In practice, these geometric aspects may not perfectly match due to approximations, so the system can be solved using measurements that describe the distance between the axes in the most relevant spatial domain. The first geometric aspect is an axis in the global coordinate system. The second geometric aspect is an axis in the first image coordinate system. The relationship between these geometric aspects is that the two axes must intersect after successful registration. The first geometric aspect is a point in the global coordinate system. The second geometric aspect is a plane in the first image coordinate system. The relationship between these geometric aspects is that after successful registration, the point must lie on a plane. The first geometric aspect is an axis in the global coordinate system. The second geometric aspect is a plane in the first image coordinate system. The relationship between these geometric aspects can be that after successful registration, the axis needs to lie in a plane. The first geometric aspect is a plane in the global coordinate system. The second geometric aspect is a plane in the first image coordinate system. The relationship between these geometric aspects can be that after successful registration, these planes are coplanar.

[0058] It will be appreciated that the above relationships can clearly be used for any number of acquired X-ray images and X-ray device positions, not just the first X-ray image.

[0059] Example 3.1 When placing the femur.

[0060] In an imaging device that acquires a first X-ray image of a proximal portion of a femur, the system uses a model of the bone to detect an imaging direction of the bone in a first image coordinate system, and an augmented reality device detects a first geometric aspect of the imaging device in a global coordinate system.

[0061] The imaging device acquires a second image of the distal portion of the femur, and the system uses the bone model to detect an imaging orientation of the bone in a second image coordinate system. The augmented reality device detects a second geometric side of the imaging device in a global coordinate system. The first and second geometric sides of the imaging device may overlap or be identical. The system estimates the geometric side of the femur using the relationship between the first and second geometric sides.

[0062] Geometric aspects include, but are not limited to, the distance between the proximal and distal ends of the femur, the ratio of femoral width to length, distal to proximal rotation, rotation angle, anteversion angle, central-cervical-diaphyseal (CCD) angle, or bone curvature.

[0063] A similar situation may apply to other long bones.

[0064] Example 3.2 When the femur is fractured and the distal and proximal portions are separated, the bone is placed.

[0065] In an imaging device that acquires a first X-ray image of a proximal femur, the system uses a bone model to detect an imaging orientation of the bone in a coordinate system of the first image, and an augmented reality device detects a first geometric side of the imaging device in a global coordinate system. The imaging device acquires a second image of a distal femur, and the system uses the bone model to detect an imaging orientation of the bone in a coordinate system of the second image. The augmented reality device detects a second geometric side of the imaging device in the global coordinate system. The system can determine the geometric side of the femur using a relationship between the first and second geometric sides.

[0066] Geometric aspects of a bone include, but are not limited to, the distance between the proximal and distal ends of the femur, the ratio of the distances (e.g., the ratio of the width to the length of the femur), the rotation of the distal part relative to the proximal part, the angle of rotation or anteversion, angles such as the central-neck-diaphyseal (CCD) angle, or the curvature of the bone.

[0067] The geometric aspect of the imaging device can be a single point, multiple points, or any two-dimensional manifold. The first and second geometric aspects of the imaging device can overlap or be identical.

[0068] This embodiment can be extended by acquiring any number of additional X-ray images and detecting the geometric aspects of the imaging device as described in the examples to improve the accuracy of determining the geometric aspects of the femur.

[0069] A similar situation may apply to other long bones.

[0070] Example 3.3 The system of Example 3.1 is used to extract a first geometric aspect from a healthy long bone, and the system of Example 3.2 is used to extract a second geometric aspect from a fractured long bone to extract information for correcting the alignment of the fracture site of the fractured long bone.

[0071] Example 3.4 This example concerns bone length estimation and guiding the physician.

[0072] The bone length may be estimated based on a series of X-ray images, as an example of a bone's geometrical aspects. The system uses additional spatial information from the augmented reality device regarding the X-ray device at the time the X-ray images were acquired. This additional spatial information may be, for example, the position of the X-ray device, the orientation of the X-ray device, a combination of the position and orientation of the X-ray device, the distance between the positions of the X-ray device, etc.

[0073] Additionally, the additional spatial information may include the position and / or orientation relative to the operating table or the patient's leg, as acquired by the augmented reality device. Based on the additional spatial information, the absolute length of the bone can be estimated by fitting a statistical model to the X-ray image and measuring the length of the estimated model. Bone length can be generalized as the ratio between two geometric aspects of the bone, such as the ratio of the bone's length along the femoral neck axis to the bone's width, without the absolute value itself being required.

[0074] Given a baseline bone length value (e.g., a value obtained from the patient's contralateral healthy bone or derived from other bone length measurement methods), the system can provide the surgeon with a guide for the correct repositioning of the fractured bone. The guide can include instructions for reducing the fracture (e.g., how to lengthen, compress, or rotate the bone to reach the baseline length). The baseline value can be absolute or relative. This baseline value can be obtained on the healthy side (e.g., the contralateral anatomical side in the case of the femur) using the same system.

[0075] Figure 1 1 is a flowchart illustrating a method according to the present disclosure. The steps described are major steps, and it will be understood that these major steps may be divided or split into multiple sub-steps. Furthermore, there may be sub-steps between these major steps.

[0076] In step S1, a first X-ray image is received, the X-ray image showing a first object, the first X-ray image being generated by an imaging device such as an X-ray imager.

[0077] In step S2, an imaging orientation of a first X-ray image is determined based on the three-dimensional representation of the first object, where the imaging orientation of the first X-ray image is associated with a coordinate system of the first X-ray image.

[0078] In step S3, information defining the pose of the imaging device when the first X-ray image was generated is received, the information including geometric aspects of the imaging device and may be provided by an augmented reality device associated with a global coordinate system.

[0079] In step S4, the position and orientation of the first object relative to the global coordinate system is determined based on registration of the coordinate system of the first X-ray image with the global coordinate system.

[0080] Between steps S3 and S4, if further tools are shown in the first X-ray image, the following steps S5, S6 and S7 may be performed.

[0081] In step S5, two degrees of freedom of the position and orientation of the tool in the global coordinate system are removed, and in step S6, two degrees of freedom of the position and orientation of the first object relative to the tool are removed. The order in which steps S5 and S6 are performed does not matter.

[0082] In step S7, geometric aspects in the coordinate system of the first X-ray image are identified. Based on the results of S5, S6, and S7, the position and orientation of the first object relative to the global coordinate system in step S4 may be further based on the relationship between the geometric aspects of the imaging device and the geometric aspects in the coordinate system of the X-ray image.

[0083] In some cases, it may be useful to process further X-ray images, in which case steps S8 to S11 may be performed.

[0084] In step S8, a second X-ray image of the second object is received, which may be generated by an imaging device such as an X-ray imager.

[0085] In step S9, an imaging direction of a second X-ray image is determined based on the three-dimensional representation of the second object, and the imaging direction of the second X-ray image can be related to the coordinate system of the second X-ray image.

[0086] In step S10, information regarding the orientation of the imaging device when the second X-ray image was generated is received, the information including geometric aspects of the imaging device and may be provided by an augmented reality device associated with a global coordinate system.

[0087] In step S11, based on the registration of the coordinate system of the second X-ray image with the global coordinate system, the position and orientation of the second object relative to the first object in the global coordinate system can be determined.

[0088] In step S12 (optional), a three-dimensional representation of at least one object seen in the X-ray image is generated.

[0089] If the first object is a first fragment of a fractured bone and the second object is a second fragment of a fractured bone, then in optional step S13, at least one aspect of the group consisting of bone length, anteversion angle, rotation angle, central-cervical-diaphyseal (CCD) angle, and bone curvature is determined.

[0090] In step 14 (optional step), a visualization is provided on the augmented reality device, the visualization being at least one of the group consisting of a three-dimensional representation of the first object, a three-dimensional representation of the second object, and a tool, and the visualization may be displayed at a position and orientation determined in the global coordinate system.

[0091] In step S15 (optional step), an insertion path of the tool or implant is determined and a visualization of the insertion path is provided on the augmented reality device, which may be displayed at a determined position and orientation in a global coordinate system.

[0092] In step S16 (optional step), predetermined position and orientation information of at least one of the first object or the second object is received and a visualization of the information may be provided on the augmented reality device.

[0093] Step S17 (optional step) identifies the current state of the bone treatment and may provide information suitable for guiding the next steps in the bone treatment.

[0094] While the embodiments have been illustrated and described in detail in the drawings and the above description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A computer program product that can be executed on the processing unit of a system, A command set for receiving a first X-ray image showing a first object generated by an imaging device. A set of instructions for determining the imaging direction of the first X-ray image, associated with the coordinate system of the first X-ray image, based on a three-dimensional representation of the first object. A set of commands provided by an augmented reality device associated with a global coordinate system, which includes the geometric aspects of the imaging device and receives information regarding the orientation of the imaging device when the first X-ray image is generated. A computer program product comprising a set of instructions for determining the position and orientation of the first object relative to the global coordinate system, based on the registration of the coordinate system of the first X-ray image and the global coordinate system.

2. The first X-ray image also displays tools, The aforementioned computer program product is A set of instructions to remove the two degrees of freedom of position and orientation of the tool in the global coordinate system. A set of instructions that removes the two degrees of freedom of the position and orientation of the first object relative to the tool, The system further includes a set of instructions for specifying the geometric aspects of the first X-ray image in the coordinate system, The computer program product according to claim 1, wherein the determination of the position and orientation of the first object with respect to the global coordinate system is further based on the relationship between the geometric aspect of the imaging device and the geometric aspect of the X-ray image in the coordinate system.

3. A command set for receiving a second X-ray image of a second object generated by the imaging device, A set of instructions for determining the imaging direction of the second X-ray image, associated with the coordinate system of the second X-ray image, based on the three-dimensional representation of the second object. A set of instructions provided by the augmented reality device, which includes or includes the geometric aspect of the imaging device and is associated with the global coordinate system, for receiving information regarding the orientation of the imaging device when the second X-ray image is generated. A set of commands that determines the position and orientation of the second object relative to the first object, based on the registration of the coordinate system of the second X-ray image and the global coordinate system. The computer program product according to claim 1, further comprising:

4. The system further includes a set of instructions for receiving a priori information about the position and orientation of the second object relative to the first object, The computer program product according to claim 3, wherein the determination of the position and orientation of the second object relative to the first object is further based on prior information of the position and orientation of the second object relative to the first object.

5. The computer program product according to claim 1, further comprising an instruction set for generating at least one of the three-dimensional representations of the first object and the three-dimensional representation of the second object.

6. The computer program product according to claim 3, wherein the first object is a first fragment of a fractured bone, the second object is a second fragment of the fractured bone, and the computer program product further includes a set of instructions for determining at least one aspect from the group consisting of bone length, anteversion angle, rotation angle, center-cervical-diaphysis (CCD) angle, and bone curvature.

7. The computer program product according to claim 1, further comprising a set of instructions for providing visualization to the augmented reality device, wherein the visualization is at least one of the group consisting of a three-dimensional representation of the first object, a three-dimensional representation of the second object, and the tools, and the visualization is displayed in a position and orientation determined in the global coordinate system.

8. The computer program product according to claim 1, further comprising a set of instructions for determining the insertion path of a tool or implant and providing a visualization of the insertion path to the augmented reality device, wherein the insertion path is displayed in a position and orientation determined in the global coordinate system.

9. The computer program product according to claim 1, further comprising a set of instructions for receiving information relating to a predetermined position and / or orientation of at least one of the first object and the second object, and providing visualization of the information on the augmented reality device.

10. The computer program product according to claim 2, further comprising a set of instructions for receiving and providing information regarding how to reach a predetermined position and / or direction of the tool or the imaging device.

11. The computer program product according to claim 1, further comprising a set of instructions for identifying the current state of bone treatment and providing information suitable for guiding the next steps in said bone treatment.

12. A system including a processing unit, wherein the processing unit is configured to execute a computer program according to any one of claims 1 to 11.

13. The system according to claim 12, further comprising an augmented reality device.

14. The system according to claim 12, further comprising an imaging device.

15. A method to support musculoskeletal surgery, A step of receiving a first X-ray image showing a first object generated by an imaging device, A step of determining the imaging direction of the first X-ray image, associated with the coordinate system of the first X-ray image, based on a three-dimensional representation of the first object. A step of receiving information about the orientation of the imaging device when the first X-ray image was generated, provided by an augmented reality device associated with a global coordinate system, including the geometric aspects of the imaging device, A step of determining the position and orientation of the first object relative to the global coordinate system based on the registration of the coordinate system of the first X-ray image and the global coordinate system. Methods that include...

Images