System and method for autonomous self-calibrating surgical robots

Abstract

The present invention provides a system and method for autonomous robotic surgery that controls the movement of a robotic device to perform surgical procedure steps. [Solution] A system and method for autonomous robotic surgery are provided, which controls the movement of a robotic device to perform surgical procedure steps, the control of movement being based on information including the spatial position and orientation of at least a portion of the robotic device. Trigger information may cause the system to pause or stop the surgical procedure steps and receive projected images. Furthermore, the projected images may be processed to determine the spatial position and orientation of an object or at least a portion of the robotic device.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, computer assistance, and robot-assisted surgery. Further, the present invention relates to a system and method for providing information about an object based on an X-ray image. In particular, the present invention relates to a system and method for automatically determining the spatial position and orientation of an object with respect to a movement space related to an anatomical structure. This method can be implemented as a computer program executable on a processing unit of the system.

[0002] Based on the foregoing aspects, a system and method for truly autonomous robotic surgery can be provided, and the system may include a self-calibrating robot.

Background Art

[0003] Computer assistance in orthopedic surgery is related to the surgeon's navigation to ensure, for example, that drilling is performed at the correct location and that implants are properly placed. This includes determining the accurate relative 3D positions and 3D orientations between surgical instruments (such as drills), implants (such as screws or nails), and anatomical structures for providing navigation instructions. Computer-assisted navigation has already been used in some areas of orthopedic surgery (such as spinal surgery), but is hardly used in other areas (especially trauma surgery). For example, in spinal surgery, computer-assisted navigation is used to accurately place pedicle screws, avoid neurovascular damage, and minimize the risk of reoperation.

[0004] However, significant problems remain when using computer assistance in orthopedic surgery. Existing navigation systems require additional procedural steps and equipment such as 3D cameras, trackers, and reference bodies. For example, in navigation-assisted spinal surgery, most current systems use optical tracking, attaching a dynamic reference system to the spine (for patient tracking) and reference bodies to instruments. In this case, both references must always be visible to the 3D camera. Such methods have several drawbacks, but are not limited to: • The system requires a time-consuming alignment procedure (at least a few minutes, and sometimes up to 30 minutes) to learn relative 3D position and orientation. • The validity and accuracy of the alignment must be continuously monitored. If the tracker moves, the alignment may need to be repeated. If the tracker's movement is not noticed, navigation instructions may become inaccurate, potentially harming the patient. • Accuracy decreases as the distance from the camera increases. • Attaching a reference system to an anatomical structure (e.g., the spine) may damage that structure.

[0005] In summary, all existing navigation systems require additional procedural steps and devices, making surgical procedures longer and more complex, as well as more expensive and prone to errors.

[0006] Robot-assisted surgical systems are gaining popularity due to the perceived accuracy they offer. However, the error-prone nature of existing navigation systems hinders truly autonomous robotic surgery. Conventional systems determine the relative 3D position and orientation between a tool (e.g., a drill) and an anatomical structure by tracking reference objects attached to the tool with cameras. By their nature, these cameras can only see externally fixed reference objects and cannot see the drill itself, which is located inside the bone. If either of the reference objects moves, or if the drill bends inside the bone, the navigation system may not detect this, providing inaccurate information and potentially harming the patient. Therefore, existing navigation technologies are not reliable enough to enable truly autonomous robotic surgery.

[0007] (i) It is desirable to have a navigation system that does not require further procedures and devices for navigation, and (ii) can determine the actual relative 3D position and orientation of surgical instruments, implants, and anatomical structures, rather than inferring from evaluations of externally fixed trackers, fiducials, or reference bodies. [Overview of the Initiative]

[0008] The present invention proposes a system and method that does not require a reference body or tracker to align multiple objects or objects that can move relative to each other in a moving space at a desired time. An object of the present invention may be to provide such alignment, i.e., determination of relative 3D position and orientation, in near real-time, possibly within a fraction of a second, based solely on the current X-ray projection image. An object of the present invention may also be to determine a particular point or curve of interest on or within an object, possibly relative to another object or in a moving space. An object of the present invention may also be to determine the relative 3D position and 3D orientation between multiple objects. In particular, an object of the present invention may be to determine the 3D position and 3D orientation of an object (e.g., drill, chisel, bone mill, reamer) or the geometric aspect of an object (e.g., drill shaft, drill tip, chisel cutting edge) in a moving space defined for anatomical structures. The moving space may be defined, for example, by a trajectory, a 1D curve, a plane, a distorted plane, a partial 3D volume, or any other manifold up to three dimensions. For example, the moving space may be defined by a drill trajectory within a vertebra.

[0009] An object of the present invention may further be to provide instructions to a surgeon or surgical robot to guide and / or restrict the movement of an object (e.g., a drill) within a movement space. The movement space does not need to be within the field of view of an X-ray image. The movement space may be determined by the system (e.g., using a neural network) based on a model of anatomical structures, or it may be predetermined by, for example, the surgeon. The predetermined movement space may also be validated by the system during surgery.

[0010] One method for determining the relative 3D position and orientation of an object in relation to the movement space is to first determine the relative 3D position and orientation of the object in relation to the anatomical structure. However, this intermediate step of determining the relative 3D position and orientation of the object in relation to the anatomical structure may not be necessary if, for example, the movement space has been determined in advance based on preoperative CT image data.

[0011] This invention teaches how to incorporate prior information to resolve ambiguities regarding the underlying 3D scenario inherent in 2D projection images such as X-ray images.

[0012] This invention may lay the foundation for truly autonomous robotic surgery. As described above, an object of the invention may be to determine the spatial position and orientation of an object relative to a moving space related to an anatomical structure, and then to guide and / or restrict the movement of the object within the moving space. For example, a robot may be instructed to drill along an implantation curve within the femur, i.e., within a moving space that encloses the volume of the drill along the implantation curve. As another example, a robotic arm may be configured so that the user can move the robotic arm only within the moving space, for example when cutting bone.

[0013] Such systems can also take into account information from other sources or sensors. For example, a pressure sensor could be integrated into the robot, allowing the drill to stop if the resistance is too high or too low.

[0014] The methods taught in this disclosure may also complement existing navigation techniques. A primary aspect of the present invention is the continuous acquisition of information from intraoperative X-rays. The present invention does not require a navigation camera or other sensors, but nevertheless, it may be combined with a navigation camera or other sensors (e.g., sensors mounted on a robot) (to increase accuracy and / or redundancy). Combining information from X-ray images with information from cameras or other sensors can improve the accuracy of determining relative spatial position and orientation, or resolve any remaining ambiguities (e.g., those that may result from occlusion). Information provided by the robot or robotic arm itself (e.g., about the movements performed) may also be considered.

[0015] As already stated above, the objective of the present invention is to determine the 3D position and orientation of an object (e.g., a drill) relative to a moving space (e.g., a drill trajectory) in near real-time, perhaps within a fraction of a second, based solely on the current X-ray image and information that can be extracted from previous X-ray images. In a truly autonomous robotic surgical system, while performing a surgical procedure step, the system itself must decide when to pause the surgical procedure step and acquire a new X-ray image (from the same and / or different imaging direction) in order to obtain a new determination of the spatial position and orientation of the object relative to the moving space. The acquisition of a new X-ray image may be triggered by several events, including, but not limited to, input from robotic sensors (e.g., pressure sensors, how far the tool has already moved), requests from a tracking-based navigation system, or exceeding a threshold in an algorithm processing the current X-ray image. Based on the information extracted from the new X-ray image, the system can continue, abort, or terminate the surgical procedure step. If the surgical procedure step is aborted, the system can perform a new plan and / or recalibrate itself, and then continue the surgical procedure step after making appropriate changes. Once a surgical procedure step is completed (e.g., a hole is drilled in the pedicle as planned), a new surgical procedure step may be initiated (e.g., the drill is removed from the patient and positioned for further drilling, or a pedicle screw is inserted into the pedicle after an appropriate change of tool).

[0016] Possible applications for applying this disclosure include all types of bone drilling, such as screw insertion into pedicles, screw insertion into sacroiliac joints, screw insertion connecting two vertebrae, and drilling for cruciate ligaments. The present invention can be used, for example, for drilling, reaming, milling, chiseling, sawing, resection, and implant positioning, and thus can support, for example, osteotomy, tumor resection, and total hip replacement.

[0017] At least one or the other of the objectives mentioned is resolved by the subject matter described in any one of the independent claims. Further embodiments of the present invention are described in their respective dependent claims.

[0018] In general, the system and method for autonomous robotic surgery according to the present invention is configured to control the movement of a robotic device, or an object held by, attached to, or controlled by a robotic device, in order to perform a surgical procedure step, and the control of movement may be based on information including the spatial position and orientation of at least a portion of the robotic device, or at least a portion of an object held by, attached to, or controlled by a robotic device. Trigger information may cause the system to pause or stop the surgical procedure step and receive a projected image. Furthermore, the projected image is processed to determine the spatial position and orientation of at least a portion of the object or robotic device. These steps may be performed as a loop. Thus, the system and method may be configured to control further movement of the robotic device, or an object held by, attached to, or controlled by a robotic device, in order to perform the next surgical procedure step.

[0019] In the context of this disclosure, the term “surgical procedure step” is intended to mean any type of movement of an object (e.g., surgical instruments such as drills or wires, implants such as intramedullary nails, bone or bone fragments, etc.) to an anatomical structure, including not only a complete set of movement but also movements divided into multiple substeps. Examples of surgical procedure steps include drilling a complete or partial hole, initiating drilling, resuming drilling that has already been started, removing a tool, rotating a tool, moving a tool to the next starting point for the next surgical step, changing tools, inserting or removing an implant, and moving an imaging device.

[0020] It should be noted that the movement of a robotic device can be understood as any movement of a part of the robotic device, such as a robotic arm or a part of a robotic arm having multiple sections, or any tool attached to the robotic device. In other words, the movement of an end effector or tool movably attached to a robotic arm can be considered movement of the robotic device.

[0021] As can be understood, the system can perform the above steps in real time without pausing the movement of the robotic device to generate and receive projected images.

[0022] According to one embodiment, trigger information may be generated based on data received from sensors of a robotic device, a navigation system, a tracking system, a camera, previous projection images, intraoperative 3D scans, a definition of the moving space, and / or any other suitable information.

[0023] According to one embodiment, deviations from the expected spatial position and orientation to the determined spatial position and orientation can be determined. Calibration information can be generated based on the determined deviations. The next movement of the robotic device can take the calibration information into consideration to improve the accuracy of the next surgical step.

[0024] According to one embodiment, the system and method may further be configured to determine the imaging direction of the next projection image. For example, if an object or structure is hidden in the current projection image generated from a particular imaging direction, it may be possible to extract more accurate information from a projection image generated from a different imaging direction. The system may be configured to suggest a suitable imaging direction or a specific pose for the imaging device. While the imaging direction is fully specified with 5 degrees of freedom, the suggestions given by the system may include more degrees of freedom to describe the movement of the imaging device to reach the suitable imaging direction (for example, based on the possibility of movement of the C-arm and / or its sub-parts).

[0025] According to one embodiment, the system and method can cause an imaging device to generate a projection image. Further or alternatively, the system and method can control the imaging device to move to a new position for generating projection images from different imaging directions. Such different imaging directions can be appropriate imaging directions proposed by the system.

[0026] Note that the control of the movement of the robotic device can be further based on at least one of the group consisting of image processing of further projection images, information from a tracking system, information from a navigation system, information from a camera, information from LIDAR, information from a pressure sensor, and calibration information.

[0027] More generally, a system and method for image-guided surgery can be provided. Such a system and method receive a model of an anatomical structure and a model of an object, and process a projection image generated by an imaging device from a certain imaging direction, the projection image including at least a part of the anatomical structure and at least a part of the object. Based on (i) the projection image, (ii) the imaging direction, (iii) the model of the object, and (iv) the model of the anatomical structure, the system and method determine the spatial position and orientation of the object with respect to the movement space. The movement space as used herein can be defined in relation to the anatomical structure.

[0028] It will be understood that the system includes a processing unit and the method can be implemented as a computer program product executable on that processing unit.

[0029] According to some embodiments, for example, when performing a surgical procedure manually, the movement of an object within a determined movement space can be monitored. Alternatively, or in addition, a robotic device may be used. The robotic device can restrict the movement of the object within a determined movement space, so that the surgical procedure is performed manually, but the robotic device acts as a safety guard, allowing movement only within the movement space. The robotic device may also be configured to actively control the movement of the object within the determined movement space.

[0030] According to some embodiments, the determination of the spatial position and orientation of an object in a moving space may be based on information received from sensors of a robotic device or on a real-time navigation system, the real-time navigation system being at least one of the group consisting of a navigation system with an optical tracker, a navigation system with an infrared tracker, a navigation system with EM tracking, a navigation system using a 2D camera, a navigation system using LiDAR, a navigation system using a 3D camera, and a navigation system including wearable tracking elements such as augmented reality glasses.

[0031] The model may be based on a (statistical) variable shape model, a surface model, a (statistical) variable appearance model, a CT scan surface model, an MR scan surface model, a PET scan surface model, an intraoperative 3D X-ray surface model, or 3D image data, the 3D image data may be a CT scan, PET scan, MR scan, or intraoperative 3D X-ray scan.

[0032] According to one embodiment, the imaging direction of the projection image can be determined by generating multiple virtual projection images from different virtual imaging directions of 3D image data, and identifying one virtual projection image from the group of virtual projection images that has the greatest similarity to the projection image.

[0033] According to further embodiments, the system and method are configured to receive a previous projection image from another imaging direction, including further parts of an anatomical structure, to detect points or lines as geometric features of the object in the previous projection image, and to detect the geometric features of the object in the projection image, wherein the geometric features of the object have not moved relative to any part of the anatomical structure between the time the previous projection image was generated and the time the projection image was generated. In such a situation, the determination of the spatial position and orientation of the object relative to moving space can be further based on the detected geometric features of the object and the knowledge that there has been no movement between the geometric features of the object and any part of the anatomical structure between the time the previous projection image was generated and the time the projection image was generated.

[0034] In a further embodiment, the system and method are further configured to receive a previous projection image from another imaging direction, including further parts of an anatomical structure, to determine the imaging direction to a first part of the object in the previous projection image, and to determine the imaging direction to a second part of the object in the projection image, wherein the object has not moved relative to any part of the anatomical structure between the time the previous projection image was generated and the time the projection image was generated. In such a situation, the determination of the spatial position and orientation of the object relative to the moving space is further based on the determined imaging direction to each part of the object and the knowledge that there has been no movement between the object and any part of the anatomical structure between the time the previous projection image was generated and the time the projection image was generated.

[0035] The determination of an object's spatial position and orientation in moving space can be based further on prior information about the spatial relationship between the object's point and a part of an anatomical structure, or on prior information about a point on the object's axis, which is defined in relation to the anatomical structure.

[0036] As used herein, “object” refers to any object that is at least partially visible in an X-ray image, or that is not visible in an X-ray image but has a known relative position and orientation to an object that is at least partially visible in an X-ray image, such as an anatomical structure, a tool, or an implant. When “object” is considered as an implant, it will be understood that the implant may already be positioned within an anatomical structure. For example, “tools” such as drills, k-wires, screws, and bone mills may also be at least partially visible in an X-ray image. In a more specific example, if “object” is bone, then “tool” may also be an implant such as a bone nail that is intended to be inserted into the bone but has not yet been inserted. It can be said that “tool” is the object to be inserted, and “object” is an object such as an anatomical structure or an implant already positioned within an anatomical structure. It should also be noted that the present invention does not require the use of any reference body or tracker, however, the use of a tracker, for example, can make the system more robust and can be used, for example, in a robotic arm.

[0037] Throughout this disclosure, the term “model” shall be understood in a very general sense. It is used to describe any virtual representation of an object (or part of an object), such as a tool or implant (or part of a tool or implant) or an anatomical structure (or part of an anatomical structure). For example, a dataset defining the shape and / or dimensions of an implant may constitute a model of the implant. As another example, a 3D representation of an anatomical structure generated during a diagnostic procedure (e.g., a 3D CT scan of a vertebra) may be a model of an actual anatomical object. It should be noted that “model” may describe a specific object, such as a specific nail or a specific vertebra of a specific patient, or a class of objects, such as vertebrae, which generally exhibit some degree of variability. In the latter case, such objects may be described, for example, by a statistical shape or appearance model. In that case, an object of the present invention may be to find a 3D representation of a specific instance from a class of objects depicted in an acquired X-ray image. For example, an object may be to find a 3D representation of a vertebra depicted in an acquired X-ray image based on a general statistical shape model of a vertebra. It is also possible to use a model that includes a discrete set of deterministic possibilities, in which case the system selects the one that best describes the object in the image. For example, there may be several implants in a database, in which case the algorithm can identify which implant is depicted in the image.

[0038] The model may be a raw 3D image of an object (e.g., a 3D CT scan of a vertebra or several vertebrae), or it may be processed 3D image data including, for example, segmentation of the object's surface. The model may also be a parameterized description of the object's 3D shape, which may also include, for example, a description of the object's surface and / or its radiation density. The model may be generated using various imaging modalities, such as one or more CT scans, one or more PET scans, one or more MR scans, mechanical sensing of the object's surface, or one or more intraoperative 3D X-ray scans, which may or may not be further processed.

[0039] Furthermore, it should be noted that the model may be a complete or partial 3D model of a real object, or it may only describe a specific geometric aspect of an object (which may have dimensions less than 3), such as the fact that the femoral or humeral head can be approximated as a ball in a 3D projection image and a circle in a 2D projection image, or that a drill has a drill shaft.

[0040] The term "3D representation" may refer to a complete or partial description of a 3D volume or 3D surface, or it may refer to selected geometric aspects such as radii, curves, planes, or angles. While the present invention may enable the determination of complete 3D information about a 3D surface or volume of an object, methods for determining only selected geometric aspects (e.g., a point representing the tip of a drill or a line representing the edge of a chisel) are also considered in the present invention. However, determining the 3D representation of a first object (e.g., an anatomical structure) may not be necessary to determine the movement space defined for the first object, and its relative 3D position and orientation to a second object.

[0041] Since X-ray imaging is a 2D imaging (projection) modality, it is generally impossible to uniquely determine the 3D pose (i.e., 3D position and orientation) of individual objects depicted in an X-ray image, and it is also generally impossible to uniquely determine the relative 3D positions and orientations between objects depicted in an X-ray image.

[0042] Since the X-ray beam originates from an X-ray source (focal point) and is detected by an X-ray detector in the image plane, the physical dimensions of an object are related to the dimensions of its projection in the X-ray image through the intercept theorem. Generally, there is ambiguity in determining the "imaging depth," which is the distance from the image plane, also referred to below as the "z-coordinate." Throughout this invention, the term "imaging direction" (also referred to as "view direction") means a 3D angle representing the direction in which a selected X-ray beam (e.g., the central X-ray beam) passes through a selected point on the object. In the example of a C-arm, the central beam of the projection imaging device is the beam between the focal point and the center of the projection plane (in other words, it is the beam between the focal point and the center of the projected image). In some cases, it may be sufficient to determine only a virtual imaging direction to a model of the object, which can be done without segmenting or detecting the object in the X-ray image, and it should be noted that the model may be raw 3D-CT image data without segmentation. For example, in a 3D-CT scan of the spine without segmentation, neither the individual vertebrae nor their surfaces are identified. In further processing, the virtual imaging direction may be used as the imaging direction for the X-ray image.

[0043] If a model of the object depicted in the X-ray image is available, this may allow for determining the imaging direction to the object. If the object is large enough and has sufficient structure, it may also be possible to determine its 3D pose. However, even if a deterministic 3D model of a known object shown in the X-ray image is available, there are cases where the imaging direction to the object cannot be determined. For example, this is particularly true for thin objects such as drills or k-wires. If the imaging depth of the drill tip is unknown, there are multiple 3D poses of the drill that result in the same or nearly the same projection in the 2D X-ray image. Therefore, in general, it may be impossible to determine, for example, the relative 3D position and 3D orientation of a drill to an implant also shown in the X-ray image.

[0044] The objective of the present invention is to determine the relative 3D position and 3D orientation of an object with a geometric shape whose imaging direction cannot be determined without further information, in relation to another object or a moving space associated with other objects.

[0045] For example, an object may include an implant having a hole. In that case, according to one embodiment of the present invention, the 3D position of a point relative to the object may be determined based on the axis of the hole in the implant. Note that the implant may be an intramedullary nail having a transversely extending through-hole for locking a bone structure with a nail. Such a hole may be provided with threads. The axis of the hole cuts into the outer surface of the bone and defines the entry point of the locking screw. In another example, a combination of a nail that may be positioned inside a long bone and a plate that may be positioned outside the long bone can be fastened together by at least one screw extending through both the hole in the plate and the hole in the nail. Here again, the entry point of the screw may be defined by the axis extending through these holes.

[0046] In a further example, the object may be considered a nail already embedded in bone, and the X-ray image also shows at least part of a tool such as a drill. In this case, the tool is at least partially visible in the first X-ray image, and the identified point is a point on the tool, for example, the tip of the tool. Even if the tool has moved relative to the object between the generation of the first X-ray image and the generation of the second X-ray image, the 3D position and orientation of the tool relative to the object can be determined based on the second X-ray image. Determining the 3D position and orientation of the drill relative to the implant in the bone at the point depicted in the second X-ray image can be useful, for example, in evaluating whether the drilling is being done in the direction (movement space) aimed at the hole for the implant through which the screw will pass when it is later embedded along the drill hole.

[0047] The 3D location of a point identified in an X-ray image can be determined in various ways. On the one hand, the 3D location of a point relative to an object can be determined based on knowledge of the location on the bone surface and the knowledge that the point is located on the bone surface. For example, when the first X-ray image is generated, the tip of a drill may be positioned on the outer surface of the bone. That point remains the same when the second X-ray image is generated, even if the drill drills into the bone. Thus, a point defined only by the tip of the drill in the first X-ray image, but which is the entry point in both X-ray images, can be the entry point.

[0048] On the other hand, the 3D position of a point relative to an object may be determined based on further X-ray images from a different viewing direction. For example, a C-arm-based X-ray system may be rotated before generating further X-ray images.

[0049] Furthermore, the 3D position of a point relative to an object can be determined based on the determination of the tool's 3D position and orientation relative to the object based on the first X-ray image. That is, if the 3D position and orientation at the time the first X-ray image is generated are already known, that knowledge can be used at a later point in time to determine the 3D position and orientation after the tool has moved relative to the object. In fact, this procedure can be repeated many times with a series of X-ray images.

[0050] If the tip of the tool is visible in the X-ray image, the determination of the tool's 3D position and orientation relative to the object may be further based on the tool tip defining further points. Note that these further points are merely points in the projected image, i.e., 2D points. However, the transition of motion between X-ray images can be determined by taking these further points into consideration, along with the known 3D position of a point, such as the entry point.

[0051] In some situations, determining the 3D position and orientation of a tool relative to an object can be more difficult. For example, at least a portion of the tool visible in an X-ray image may be rotationally symmetric, such as a drill rotating during the generation of the X-ray image. Nevertheless, according to one embodiment of the present invention, the 3D position and orientation of a tool relative to an object can be determined with at least sufficient accuracy. For example, when considering an elongated tool or implant such as a drill or k-wire, a single projection may not provide enough detail to determine the orientation of the tool in 3D space, and those orientations may consequently result in similar or identical projections. However, when comparing more than one projection image, a particular orientation is more likely to exist, and that orientation can be assumed. Furthermore, additional aspects, such as the visible tip of the tool, may also be taken into consideration.

[0052] In another example, when generating an X-ray image showing an object and a tool together, the tool may be partially hidden. The tip of the tool may be hidden by an implant, or the shaft of a drill may be mostly hidden by a tube, which protects surrounding soft tissue from damage during bone drilling. In these cases, a third X-ray image, generated from a different viewing direction than the previous image, may be received. Such a third X-ray image may provide additional information beyond what can be obtained from the image generated in the main viewing direction. For example, the tip of the tool may be visible in the third X-ray image. The axis of the tool visible in the second X-ray image defines a plane oriented toward the focal point of the X-ray imaging device when generating the second X-ray image, and consequently, the 3D position of the tip can be determined even if it is not visible in the second X-ray image, since the axis of the tool visible in the second X-ray image defines a plane oriented toward the focal point of the X-ray imaging device. Furthermore, the tip of the tool can be thought of as defining a line oriented toward the focal point of the X-ray imaging device when generating the third X-ray image. The line defined by the tip, i.e., the point visible in the third X-ray image, cuts a plane in 3D space defined based on the second X-ray image. It will be understood that the second and third X-ray images are aligned, for example, by determining the imaging direction of the object in both images.

[0053] It should be noted that the first X-ray image mentioned in the previous paragraph, which provides prior information, may be replaced, for example, with a model of the anatomical object in the form of a segmented CT scan of the anatomical object, or a statistical model of the surface of the anatomical object. Furthermore, in that case, an identified point of another object (e.g., the tip of a drill) has a known 3D distance to the surface of the anatomical object (e.g., the tip of the drill touches the bone).

[0054] Based on the processed X-ray images, the device may be configured to provide instructions to the user or to automatically perform corresponding actions. In particular, the device may be configured to compare the determined 3D position and orientation of the tool relative to the object with the expected or intended 3D position and orientation. Monitoring during drilling is possible, as well as determining the appropriate orientation at the start of drilling. For example, during drilling, the device may evaluate whether the drilling direction will eventually hit the target structure and decide to correct the drilling direction as necessary. When providing instructions or performing actions autonomously, the device may take into account at least one of the already performed drilling depth, object density, drill diameter, and drill stiffness. It will be understood that tilting the drill during drilling may cause the drill to bend or the drill axis to shift, depending on the properties of the surrounding material, such as bone. These aspects, which can be expected to some extent, may be taken into account by the device when providing instructions or performing actions autonomously.

[0055] One possible solution proposed by EP19217245 is to use prior information about the imaging depth. For example, from previous X-ray images acquired from different imaging directions (indicating the direction in which the X-ray beam passes through the object), it can be determined that the tip of the k-wire is on the trochanter, and therefore the imaging depth of the tip of the k-wire relative to another object can be constrained. This may be sufficient to resolve ambiguity about the 3D position and 3D orientation of the k-wire relative to another object in the current imaging direction.

[0056] 3D alignment of two or more X-ray images Another possible solution is to use two or more X-ray images acquired from different imaging directions and align these images. The more different the imaging directions (e.g., AP images and ML images), the more useful the additional images may be in determining the 3D information. Image alignment proceeds based on determining the imaging direction to an object depicted in an image, whose 3D model is known and should not move between images. As mentioned above, the most common approach in the art is to use a reference body or tracker. However, it is generally preferable not to use a reference body because it simplifies both product development and system use. If the movement of the C-arm is known precisely (e.g., if the C-arm is electronically controlled), image alignment may be possible based solely on these known C-arm movements.

[0057] Furthermore, there are many scenarios in which such rigid bodies do not exist in the X-ray image. For example, when determining an entry point for implanting a nail, the implant is not present in the X-ray image. The present invention teaches a system and method that enables 3D registration of multiple X-ray images in the absence of a single rigid body of known geometric shape that would enable intrinsically accurate 3D registration. The proposed method involves using a combination of features of two or more objects or features of at least two or more parts of one object, which, in themselves, do not enable intrinsically accurate 3D registration, but when combined, enable such registration and / or restrict the permissible C-arm movement between image acquisitions (e.g., only rotation around a specific axis of the X-ray imaging device, such as the C-arm axis, or translation along a specific axis may be permitted). The objects used for registration may be artificial objects of known geometric shape (e.g., a drill or k-wire), or parts of anatomical structures. The objects or parts of objects may be approximated using a simple geometric model (e.g., the femoral head may be approximated by a ball), or only specific features of them (a single point, e.g., the tip of a k-wire or drill) may be used. The object features used for alignment must not move between image acquisitions, and if such a feature is a single point, only the fact that this point does not move is required. For example, if the tip of a k-wire is used, the tip must not move between images, but the tilt of the k-wire may change between images.

[0058] According to one embodiment, the X-ray images can be aligned, each showing at least a portion of the object. The first X-ray image may be generated in a first imaging direction at a first position of the X-ray source relative to the object. The second image may be generated in a second imaging direction at a second position of the X-ray source relative to the object. The two such X-ray images can be aligned with a model of the object and based on at least one of the following conditions: - A point with a fixed 3D position relative to the object can be defined and / or detected in both X-ray images, for example, identifiable in both X-ray images. Note that a single point may suffice. Furthermore, note that this point may have a known distance to the object's structure, such as its surface. - Two identifiable points with fixed 3D positions relative to the object are present in both X-ray images. - Parts of further objects with fixed 3D positions are visible in both X-ray images. In such cases, a model of the further object may be used when aligning the X-ray images. Even a single point can be considered part of the further object. - Between the acquisition of the first and second X-ray images, the only movement of the X-ray source relative to the object is translation. - Between the generation of the first and second X-ray images, the only rotation of the X-ray source is rotation around an axis perpendicular to the imaging direction. For example, the X-ray source may rotate around the C-axis of a C-arm-based X-ray imaging device.

[0059] It will be understood that the alignment of X-ray images based on an object model can be more accurate when combined with one or more of the aforementioned conditions.

[0060] According to one embodiment, a point having a fixed 3D position relative to an object may be a point on a further object, and further movement of the object is permitted as long as that point remains fixed. It will be understood that the fixed 3D position relative to an object may be on the surface of the object, i.e., a point of contact, or it may be a point at a defined distance (greater than zero) from the object. This can be at a certain distance from the surface of the object (possibly outside or inside the object) or at a certain distance from a specific point on the object (for example, the center of the ball if the object is a ball).

[0061] According to one embodiment, a further object having a fixed 3D position relative to an object may be in contact with the object or at a defined distance from it. Note that the orientation of the further object relative to the object may be fixed or variable, and the orientation of the further object may be changed by rotation and / or translation of the further object relative to the object.

[0062] It will also be understood that X-ray image alignment may be performed using three or more objects.

[0063] According to various embodiments, the following is an example of enabling image alignment (without a reference object): 1. Approximate the femoral head or artificial femoral head (as part of a hip joint implant) using a ball (object 1) and a k-wire or drill tip (object 2), while simultaneously constraining the allowable C-arm movement between images. 2. Approximate the diaphysis or vertebral body using a cylindrical object (Object 1) and use a k-wire or drill tip (Object 2). The permissible C-arm movement may or may not be constrained between images. 3. Use a ball (Object 1) approximation of the femoral head or artificial femoral head (as part of a hip joint implant) and a cylinder (Object 2) approximation of the femoral shaft. There is no need to restrict the allowable C-arm movement between images. 4. A guide rod (the guide rod has a stopper to prevent excessive insertion) or a k-wire fixed within the bone is used, while simultaneously restricting the permissible C-arm movement between images. In this case, only one object is used, and this method is embodied by the constrained C-arm movement between images. 5. Use an approximation with a guide rod or k-wire fixed within the bone (object 1) and a ball of the femoral head (object 2).

[0064] It should be noted that this method can also be used to improve the accuracy of alignment or to verify other results. That is, when aligning an image using multiple objects or at least multiple parts of an object (one or more of which may enable 3D alignment on their own), and possibly constraining the simultaneously allowed movement of the C-arm, this over-determination may improve the accuracy of alignment compared to not using the proposed method. Alternatively, the image may be aligned based on a subset of available objects or features. Such alignment may be used to verify the detection of the remaining objects or features (that were not used for alignment), or to enable the detection of movement between images (e.g., whether the tip of the mouthpiece has moved).

[0065] A further embodiment of this technique may involve aligning two or more X-ray images depicting different (but potentially overlapping) parts of an object (e.g., one X-ray image showing the proximal part of a femur and another showing the distal part of the same femur) by jointly fitting a model to all available X-ray projections while restricting the allowed C-arm movement between X-ray images (e.g., only translation is allowed). The fitted model may be a complete or partial 3D model (e.g., a statistical shape or appearance model), or a reduced model describing only specific geometric aspects of the object (e.g., axes, planes, or the location of selected points).

[0066] As will be explained in detail below, the 3D reconstruction of an object may be determined based on aligned X-ray images. It will be understood that X-ray image alignment may be performed and / or enhanced based on the 3D reconstruction of an object (or at least one of several objects). The 3D reconstruction determined based on aligned X-ray images may be used for further X-ray image alignment. Alternatively, the 3D reconstruction of an object may be determined based on a single or first X-ray image and a 3D model of the object, and then used when aligning a second X-ray image with the first X-ray image.

[0067] In general, X-ray image registration and / or 3D reconstruction can be advantageous in the following situations: I am interested in determining the anterior tilt angle in the femur. I am interested in determining the torsional angle of the tibia or humerus. I am interested in determining the CCD angle between the femoral head and the femoral shaft. I am interested in determining the lordosis of long bones. I am interested in determining bone length. • I am interested in determining the optimal implant placement site in the femur, tibia, or humerus.

[0068] Below are some examples of object combinations for illustrative purposes. Object 1 is the humeral head, and the location is the tip of a mouth opener or drill. Object 1 is a vertebra, and the point is the tip of an opening instrument or drill placed on the surface of the vertebra. Object 1 is the tibia, and the location is the tip of the mouth opener. Object 1 is the tibia, and Object 2 is the fibula, femur, talus, or another bone of the foot. Object 1 is the proximal part of the femur, and Object 2 is an opening device located on the surface of the femur. Object 1 is the distal part of the femur, and Object 2 is an opening device located on the surface of the femur. Object 1 is the distal part of the femur, Object 2 is the proximal part of the femur, at least one X-ray image depicts the distal part of the femur, at least one X-ray image depicts the proximal part of the femur, and a further object is an opening device placed on the proximal part of the femur. Object 1 is the ilium, object 2 is the sacrum, and the location is the mouth opener or the tip of a drill. Object 1 is an intramedullary nail embedded in a bone, and Object 2 is the bone itself. Object 1 is an intramedullary nail embedded in the bone, Object 2 is the bone, and the location is the tip of a sub-implant such as an opening device, drill, or locking screw.

[0069] For example, if a 3D model of an anatomical structure in the form of a 3D CT scan is available, the imaging direction to this anatomical structure can be determined by matching the model to an X-ray image. For this purpose, digitally reconstructed radiographs (DRRs) are calculated for multiple imaging directions, and the DRR that best matches the X-ray image is adopted to determine the imaging direction.

[0070] To determine the imaging direction, it may not be necessary to constrain the DRR to the anatomical structure of interest, and therefore, it may not be necessary to segment the anatomical structure of interest in the model. When evaluating the best match between the DRR and the X-ray image, it may be possible to highlight the anatomical structure of interest (e.g., with appropriate weighting) if, for example, the tip of the drill depicted in the X-ray image is pointing towards the structure of interest. However, in general, it may not be essential to detect the anatomical structure of interest in the X-ray image.

[0071] As explained in the previous paragraph, two X-ray images can be aligned by determining their respective imaging directions. If both images depict an object in which a point in each image that has not moved relative to the anatomical structure between the acquisition of the two images can be detected (e.g., a surgeon pointing the tip of a drill towards a bone surface, with the drill being allowed to be tilted), then the 3D position of that point can be determined relative to a 3D model. First, the two imaging directions for the two X-ray images are determined. Next, the point (e.g., the midpoint) on the shortest line connecting the epipolar lines passing through each point (e.g., the drill tip position) is calculated. This point determines the 3D position of the point (e.g., the drill tip) relative to the 3D model and therefore relative to the defined moving space. The distance between the epipolar lines can be used for verification. The moving space is defined with respect to the anatomical structure of interest, but does not need to be within the anatomical structure of interest, nor does it need to be within the field of view of the X-ray image. If prior information about the spatial position relative to the model or the moving space is available, this information can be used to improve the accuracy of the alignment. This allows for the mutual optimization of image alignment and the determination of the spatial position of points that may deviate from the points defined above. Note that if the object allows for the detection of lines that do not move between the acquisition of two X-ray images (e.g., the tip of a chisel blade), a similar procedure is possible that yields an epipolar plane instead of an epipolar line. However, the epipolar plane does not offer an option for verification.

[0072] If both X-ray images depict an object (e.g., a drill) that did not move relative to the anatomical structure between the acquisition of the two images, then joint optimization of the imaging direction to the anatomical structure of interest and the imaging direction to the object can be performed. This is applicable when a robot is performing drilling.

[0073] If prior information is available about the location of an object relative to an anatomical structure (e.g., the tip of a drill on a bone surface), this can be used to 3D reconstruct the anatomical structure (e.g., the bone surface must contain this location).

[0074] All the procedures described can also be applied to aligning more than two X-ray images.

[0075] Calculation of 3D representation / reconstruction When two or more radiographs are aligned, they can be used to compute a 3D representation or reconstruction of an anatomical structure that is at least partially depicted in the radiographs. According to one embodiment, this is carried out in line with the approach proposed by P. Gamage et al., “3D reconstruction of patient specific bone models from 2D radiographs for image-guided orthopedic surgery,” DOI: 10.1109 / DICTA.2009.42. In the first step, the features of the bone structure of interest (usually characteristic bone edges, which may include the outer bone contour and some characteristic internal edges) are determined in each radiograph, perhaps using a neural network trained for segmentation. In the second step, the 3D model of the bone structure of interest is deformed so that its 2D projection fits the features determined in the first step (e.g., characteristic bone edges) in all available radiographs. While Gamage et al. use a general-purpose 3D model of the anatomical structure of interest, other 3D models, such as statistical shape models, may also be used. It should be noted that this procedure requires not only the relative field of view between images (provided by image alignment) but also the imaging direction of one of the images. This direction may be known (for example, because the surgeon was instructed to acquire images from a specific viewing direction, e.g., anterior-posterior (AP) or medial-lateral (ML)) or it may be estimated based on various methods (e.g., by using LU100907B1 or as described above). While a more accurate relative field of view between images may increase the accuracy of 3D reconstruction, the accuracy of determining the imaging direction of one of the images may not be a critical factor.

[0076] The accuracy of the determined 3D representation can be improved by incorporating prior information about the 3D location of one or more points on the bone structure of interest, or even a portion of its surface. For example, in a 3D reconstruction of a femur with a nail embedded, a k-wire may be used to indicate a specific point on the surface of the femur in an X-ray image. From a previous procedural step, the 3D location of this indicated point in the coordinate system given by the embedded nail may be known. In that case, this knowledge can be used to more accurately reconstruct the 3D surface of the femur. If such prior information about the 3D location of a specific point is available, 3D reconstruction based on a single X-ray image is also possible. Furthermore, if the implant (such as a plate) matches the shape of a portion of the bone and is positioned in that matching portion of the bone, this information can be used in the 3D reconstruction.

[0077] As an alternative approach, 3D reconstruction of an object (e.g., a bone) may be performed without prior image registration; that is, image registration and 3D reconstruction may be performed jointly. This disclosure teaches that accuracy can be increased and ambiguity resolved by constraining the permissible C-arm movement and / or utilizing easily detectable features of another object (e.g., a drill or k-wire) present in at least two of the images on which the joint registration and reconstruction is based. Such easily detectable features could be, for example, the tip of a k-wire or drill on the surface of the object being reconstructed or at a known distance therefrom. This feature must not move between image acquisitions. In the case of a k-wire or drill, this means that the tilt of the instrument itself may change, as long as its tip remains in place. If more than two images are used for such reconstruction, reconstruction without prior image registration may perform better. It should be noted that joint image registration and 3D reconstruction may generally perform better than methods that perform registration first, because joint registration and 3D reconstruction allows for the joint optimization of all parameters (i.e., for both registration and reconstruction). This is especially true in cases of over-determination, for example, when reconstructing the 3D surface of bone using prior information about the 3D location of embedded nails or plates and points on the surface.

[0078] For joint image registration and 3D reconstruction, a first X-ray image showing a first portion of a first object is received, and the first X-ray image is generated in a first imaging direction and at a first position of the X-ray source relative to the first object; and at least a second image showing a second portion of the first object is received, and the second X-ray image is generated in a second imaging direction and at a second position of the X-ray source relative to the first object. By using a model of the first object, the projection of the first object in the two X-ray images can be jointly matched, and thus the spatial relationships of the images can be determined by deforming the model and adapting it to match the appearance in the X-ray images. The results of such joint registration and 3D reconstruction may be enhanced by at least one point having a fixed 3D position relative to the first object, which is identifiable and detectable in at least two of the X-ray images (it will be understood that it is also possible to register more than two images while improving the 3D reconstruction). Furthermore, at least a portion of a second object having a fixed 3D position relative to the first object can be taken into consideration, and based on the model of the second object, at least a portion of the second object can be identified and detected in the X-ray image.

[0079] It should be noted that the first and second parts of the first object may overlap, which improves the accuracy of the results. For example, the so-called first and second parts of the first object may both be the proximal part of the femur, but because the imaging directions are different, at least the appearance of the femur will be different in the images.

[0080] Determination of embedding curves and / or entry points The object of the present invention may be to determine an implantation curve or path along which an implant, such as a nail or screw, is inserted and embedded in the bone, and / or to determine an entry point, which is the point where the surgeon opens the bone to insert the implant. Thus, the entry point is the intersection of the implantation curve and the bone surface. The implantation curve may be a straight line (or axis), or it may be curved, since the implant (e.g., a nail) has a curve. It should be noted that the optimal location of the entry point may depend on the location of the implant and the bone fracture, i.e., whether the fracture is distal or proximal.

[0081] There are various situations in which it is necessary to determine the implantation curve and / or entry point. In some cases, particularly when sufficient anatomical reduction has not yet been performed, only the entry point may be determined. In other cases, the implantation curve is determined first, and then the entry point is determined by determining the intersection of the implantation curve and the bone surface. In yet another case, the implantation curve and entry point are determined together. All examples of these cases are described in this invention.

[0082] Generally, according to one embodiment, a 2D X-ray image is received, which indicates the region of interest for the surgery. In the X-ray image, a first point associated with the structure of interest and an implantation path within the bone for the implant intended to be implanted are determined, and the implantation curve or path has a predetermined relationship with respect to the first point. The entry point for inserting the implant into the bone lies on the implantation path. It will be understood that the first point does not have to be the entry point.

[0083] Based on 3D reconstruction of the bone, the system can also assist in selecting an implant and calculating its in-bone position (implantation curve) (i.e., entry point, insertion depth, rotation, etc.) to ensure the implant is sufficiently far from narrow spots in the bone. Once the entry point is selected, the system can calculate a new ideal in-bone position based on the actual entry point (if the implant is already visible in the bone). The system can then update the 3D reconstruction to take into account the actual position of the bone fragments. The system can also calculate and display the projected position of sub-implants that have not yet been implanted. For example, in the case of a medulla nail, the projected position of the neck screw / blade can be calculated based on a complete 3D reconstruction of the proximal femur.

[0084] Freehand locking procedure For example, when considering the implantation of a screw to lock a bone nail, based on the aforementioned general determination of the location and implantation path in a 2D X-ray image, the following conditions may be met regarding a given relationship between the implantation path and its location: when the structure of interest is the hole of the implant, the hole has a given axis, its location is associated with the center of the hole, and the implantation path may point in the direction of the hole's axis. The hole may be considered a moving space.

[0085] As a possible application example, we describe an exemplary workflow for a freehand locking procedure in which an implant is locked by embedding a screw into the implant hole. In one embodiment, the imaging direction to an already embedded nail is determined by an X-ray image, thereby determining the embedding curve. Here, the embedding curve is a straight line (axis) along which the screw is embedded. A 3D reconstruction of the bone surface (at least in the vicinity of the embedding curve) can be performed with respect to the already embedded nail (i.e., within the coordinate system given by the nail). This proceeds as follows: At least two X-ray images are acquired from different viewing directions (e.g., one is an AP or ML image and the other is an image taken from an oblique angle). The X-ray images are classified and aligned by a neural network, for example using the embedded nail, and the bone contour is segmented in all images, possibly by the neural network. A 3D reconstruction of the bone surface is possible according to the 3D reconstruction procedure outlined above. The intersection of the embedding curve and the bone surface determines the 3D position of the entry point for the nail. Since the viewing direction of the X-ray image can be determined, it becomes possible to indicate the position of the entry point in a given X-ray image.

[0086] The accuracy of this procedure may be increased by incorporating the known 3D location of at least one point on the bone surface relative to the nail. Such knowledge may be obtained by combining the procedure of the present invention with the freehand locking procedure taught in EP19217245. A possible technique is to use EP19217245 to obtain the entry point of the first lock hole and make it a known point on the bone surface. This known point can be used in the present invention for 3D reconstruction of the bone and subsequent determination of the entry points of second and further lock holes. The point on the bone surface may also be identified, for example, by the tip of the drill touching the bone surface. Accuracy may be increased if the point is identified in more than one X-ray image taken from different imaging directions.

[0087] Determining the entry point for embedding the nail in the femur. When considering the implantation of a nail into the femur, based on the aforementioned general determination of the first site and implantation path in a 2D X-ray image, at least one of the following conditions may be satisfied for a given relationship between the implantation path and the first site: When the structure of interest is the femoral head, the first site is associated with the center of the femoral head and, consequently, can be located proximal to the proximal extension of the implantation pathway, i.e., proximal to the entry point in the X-ray image. When the structure of interest is the narrow portion of the femoral neck, the first point is associated with the center of the cross-section of the narrow portion of the femoral neck, and the proximal extension of the implantation path may be closer to the first point than to the outer surface of the femoral neck in the narrow portion. When the structure of interest is a narrow portion of the femoral shaft, the first point is associated with the center of the cross-section of the narrow portion at the proximal end of the femoral shaft, and the implantation path may be closer to the first point than to the outer surface of the femoral shaft in the narrow portion. When the structure of interest is the narrowest part of the femoral shaft, the first point is associated with the center of the cross-section of the narrowest part, and the first point may be located on the implantation path.

[0088] In some embodiments, the structure of interest does not need to be fully visible in the X-ray image. It may be sufficient if only 20 to 80 percent of the structure of interest is visible in the X-ray image. Depending on the specific structure of interest, i.e., whether the structure of interest is the femoral head, femoral neck, femoral shaft, or another anatomical structure, at least 30 to 40 percent of the structure must be visible. As a result, for example, even if the center of the femoral head itself is not visible in the X-ray image, i.e., it is outside the imaging area and only 20 to 30 percent of the femoral head is visible, it may still be possible to identify the center of the femoral head. The same is possible for the narrowest part of the femoral shaft, even if the narrowest part is outside the imaging area and only 30 to 50 percent of the femoral shaft is visible.

[0089] To detect points of interest in an image, a neural segmentation network can be used to classify whether each pixel is a potential major point. A neural segmentation network can be trained on a 2D Gaussian heatmap where the center is the true major point. Gaussian heatmaps may be rotationally invariant, or they may be directional if uncertainty in a particular direction is acceptable. To detect points of interest outside the image itself, one possible technique is to segment further pixels outside the original image using all the information contained within the image itself, in order to allow extrapolation.

[0090] An exemplary workflow for determining the entry point for implanting an intramedullary or head nail in the femur is presented. According to one embodiment, first, the projection of the implantation curve is determined relative to the X-ray image. In this embodiment, the implantation curve is approximated by a straight line (i.e., the implantation axis). As a first step, it may be checked whether the current X-ray image meets the requirements necessary for determining the implantation axis. These requirements may include image quality, sufficient visibility of specific areas of anatomical structures, and at least a generally appropriate field of view (ML) to the anatomical structures. Furthermore, the requirements may include whether the above conditions are met. These requirements may be checked by an image processing algorithm, possibly utilizing a neural network. Furthermore, where applicable, the relative positions of bone fragments may be determined and compared with their desired positions, and based on this, it may be determined whether these bone fragments are sufficiently well positioned (i.e., whether the anatomical reduction is sufficiently well performed).

[0091] More specifically, the above conditions are described as follows: The implantation axis is determined by one point and direction associated with at least two anatomical landmarks (for example, these may be the center of the femoral head and the narrowest part of the femoral shaft). As previously mentioned, landmarks may be determined by a neural network even if they are not visible in the X-ray image. The feasibility of the proposed implantation axis can be checked by determining the distance from the proposed implantation axis to various landmarks on the X-ray-visible bone contour. For example, the proposed implantation axis should pass near the center of the narrowest part of the femoral neck, i.e., it should not get too close to the bone surface. If the moving space corresponding to the volume of the bone nail is considered to extend along the implantation axis, there should be no collision with the bone surface. Under such conditions, it may not be possible to obtain an X-ray image from a suitable imaging direction, and it may be necessary to obtain another X-ray image from a different imaging direction. By determining the implantation curve in another X-ray image from a different viewing direction, different implantation axes can be obtained, and therefore different entry points can be obtained. The present invention also teaches how to adjust the imaging device to obtain an X-ray image from a suitable direction. Note that both embedded axes may be located within the moving space.

[0092] It should be noted that implants may have curvature, meaning that a straight embedding axis may only approximate the projection of the inserted implant. Alternatively, the present invention may also determine an embedding curve that more closely follows the 2D projection of the implant, based on a 3D model of the implant. Such a method may determine the embedding curve, and therefore the migration space, using multiple points associated with two or more anatomical landmarks.

[0093] The projection of the implantation axis determines the implantation plane in 3D space (or, more generally, the projection of the implantation curve determines the 2D manifold in 3D space). The entry point is obtained when this implantation plane intersects with another bone structure that can be approximated by a line and is known to contain the entry point. In the case of the femur, such a bone structure is the peritrochanter, which is narrow and straight enough to be approximated by a line, and it is assumed that the entry point lies on it. Note that other locations for the entry point may be possible, such as on the piriform fossa, depending on the implant.

[0094] The peritrochanteric region may be detectable on lateral radiographs. Alternatively, or in addition, another point identifiable in the image (e.g., the tip of the k-wire or other opening tool depicted) can be used, in which case some prior information about its position relative to the entry point is known. In the case of the femur, one example is when it is known that the tip of the k-wire is located on the peritrochanteric region, which can be determined by palpation and / or because previously acquired radiographs from different field angles (e.g., AP) constrain the position of the k-wire tip to at least one dimension or degree of freedom.

[0095] There may be at least three ways to utilize such prior information about the tip of the k-wire (or some other opening instrument) relative to the entry point. The simplest method may be to use the orthogonal projection of the k-wire tip onto the projection of the implantation axis. In this case, after repositioning the k-wire tip based on the information in the ML image, and possibly after acquiring a new ML image after repositioning, it may be necessary to check whether the k-wire tip is still on the desired structure (peritrochanter) in subsequent X-ray images acquired from a different angle (e.g., AP). Another possible method may be to estimate the angle between the projection of the structure (which may not be identifiable in the ML image) and the projection of the implantation axis based on anatomical prior information, and project the k-wire tip obliquely onto the projection of the implantation axis at this estimated angle. Finally, a third possible method may be to use aligned pairs of AP and ML images to calculate in the ML image the intersection of projected epipolar lines defined by connecting the k-wire tip and the focus of the AP image to the projected implantation axis. Once the entry point is obtained, the implantation axis in 3D space is also determined.

[0096] Alternatively, by performing a partial 3D reconstruction of the proximal femur, the bone structure (in this case, the peritrochanteric region) can be located, where the intersection with the implantation plane determines the entry point. According to one embodiment, this 3D reconstruction proceeds as follows, based on two or more X-ray images from different viewing directions, at least two of which include the k-wire: A characteristic bone edge of the femur (including at least the bone contour) is detected in all X-ray images. Furthermore, the femoral head is found, approximated by a circle, and the tip of the k-wire is detected in all X-ray images. Here, the images may be aligned using the above method based on the characteristic bone edge, the approximated femoral head, the tip of the k-wire, and the movement of the constrained C-arm. After image alignment, a 3D surface including at least the trochanteric region may be reconstructed. The accuracy of the 3D reconstruction can be increased by utilizing prior information about the distance from the bone surface to the tip of the k-wire (e.g., which may be obtained from AP images). Various alternatives to this procedure are possible and will be described in the detailed description of the embodiments.

[0097] In the above method, the implantation curve is determined using 2D X-ray images, and then various alternatives for obtaining the entry point are described. Alternatively, the entire procedure (i.e., determination of the implantation curve and entry point) may be based on a 3D reconstruction of the proximal femur (or distal femur if retrograde nails are used) including a sufficient portion of the diaphysis. Such a 3D reconstruction may also be based on multiple X-ray images that have been aligned using the method described above. For example, alignment may use a ball-shaped approximation of the femoral head and a cylindrical or mean diaphysis shape approximation of the diaphysis. Alternatively, joint optimization and alignment determination and bone reconstruction (including the surface and possibly internal structures such as the pulposus and internal cortex) may be performed. Once the 3D reconstruction of the relevant portion of the femur is obtained, the 3D implantation curve can be fitted by optimizing the distance between the implant surface and the bone surface. The intersection of the 3D implantation curve and the already determined 3D bone surface becomes the entry point.

[0098] The position and orientation of the embedding curve relative to a 2D X-ray image are determined based on a first point, the embedding curve includes a first region within the bone having a first distance to the bone surface and a second region within the bone having a second distance to the bone surface, where the first distance is smaller than the second distance, and the first point lies on a first identifiable structure of the bone and at a certain distance from the first region of the embedding axis. A second point may be used, which lies on an identifiable structure of the bone and at a certain distance from the second region of the embedding curve. Furthermore, the position and orientation of the embedding curve are further determined based on at least one additional point, where at least one additional point lies on a second identifiable structure of the bone and lies on the embedding curve. The movement space may be defined by the embedding curve.

[0099] Determining the entry point for implanting the nail into the tibia. As explained in the "Calculation of 3D Representation / Reconstruction" section above, based on joint alignment and 3D reconstruction, the entry point for implanting the intramedullary nail in the tibia can be determined.

[0100] In one embodiment, accuracy is increased and ambiguity is resolved by requiring the user to position an opening instrument (e.g., a drill or k-wire) on the surface near any point, ideally a possible entry point, on the proximal part of the tibia. The user acquires a lateral image and at least one AP image of the proximal part of the tibia. The 3D reconstruction of the tibia can be calculated by jointly fitting a statistical model of the tibia to the projection of all X-ray images, taking into account the fact that the tip of the opening instrument does not move between images. Accuracy can be further increased by requiring the user to acquire two or more images from different (e.g., roughly AP) imaging directions, and possibly another (e.g., lateral) image as well. Over-determination allows for the detection of possible movement of the tip of the opening instrument and / or verification of the detection of the tip of the opening instrument.

[0101] Based on a 3D reconstruction of the tibia, the system may determine the entry point, for example, by identifying the entry point on the mean shape of a fitted statistical model. It should be noted that such guidance for finding the entry point for an antegrade tibial nail based solely on imaging (i.e., without palpation) may allow surgeons to perform a suprapatellar approach, which is generally preferred but has the disadvantage that palpation of the bone at the entry point is conventionally impossible.

[0102] Determining the entry point for implanting a nail into the humerus. A further application of the image registration and reconstruction techniques proposed above could be the determination of the entry point for implanting an intramedullary nail in the humerus.

[0103] In general, to support radiographic humeral surgery to achieve the aforementioned objectives, a system including a processing unit for processing radiographic images may be used. When a software program product is run on the processing unit, the system can be made to perform a method including the following steps: First, a first radiographic image is received, generated in a first imaging direction and showing the proximal part of the humerus; and a second radiographic image is received, generated in a second imaging direction and showing the proximal part of the humerus. These images may include the proximal part of the humeral shaft, as well as the humeral head with its articular surface, and further, the glenoid fossa, i.e., the complementary articular structure of the shoulder. Note that the second imaging direction is usually different from the first imaging direction. Next, (i) the first and second X-ray images are aligned, (ii) an approximation of at least a portion of the 2D contour of the humeral head is determined in both images, (iii) a 3D approximation of the humeral head is determined based on the approximated 2D contour and the alignment of the first and second images, and (iv) the 2D image coordinates of at least three different points in total in the first and second X-ray images are determined. Finally, an approximation of the anatomical neck is determined as a curve on the 3D approximation of the humeral head based on at least three determined points. Note that the at least three determined points do not need to lie on the determined curve. If further points on the anatomical neck that are not coplane with the first three points can be determined, a more accurate approximation of the anatomical neck can be determined. This makes it possible to determine the rotational position of the anatomical neck, and therefore the rotational position of the humeral head around the shoulder joint axis. Another method for determining the rotational position around the joint axis may be to detect the positions of the greater tubercle and / or lesser tubercle, provided that at least one of the two is in a fixed position relative to the proximal bone fragment. Another alternative might involve using preoperatively acquired 3D information (e.g., CT scans) to generate a 3D reconstruction of the proximal bone fragment based on intraoperative X-ray images. This method can be combined with the previously described method.

[0104] According to one embodiment, at least a portion of the 2D contour of the humeral head may be an approximation of a 2D circle or a 2D ellipse. Furthermore, the 3D approximation of the humeral head may be a 3D ball or a 3D ellipse. The approximation of the anatomical neck may be a circle or an ellipse in 3D space.

[0105] According to one embodiment, further X-ray images are received, and an approximation of the humeral diaphysis axis can be determined from at least two X-ray images selected from the group consisting of a first X-ray image, a second X-ray image, and further X-ray images. Based on the approximated humeral diaphysis axis from at least two X-ray images and the alignment of the first and second X-ray images, an approximation of the 3D diaphysis axis of the humerus can be determined.

[0106] Subsequently, according to one embodiment of the disclosed method, the entry point and / or displacement of the proximal fragment of the fractured humerus can be determined based on an approximated anatomical neck, an approximated 3D diaphyseal axis, and / or an approximated glenoid fossa of the humeral joint. As a result, the embedding curve in the proximal fragment can be determined based on the entry point and the displacement of the humeral head. Furthermore, information for repositioning the proximal fragment can be provided.

[0107] According to one embodiment, at least two X-ray images are aligned, and these two X-ray images may be two of a first X-ray image, a second X-ray image, and a further X-ray image. The X-ray images may be aligned based on a model of the humeral head and based on one further point having a fixed 3D position relative to the humeral head, which is identified and detected in at least two X-ray images. The one further point may be the tip of the instrument and may be on the articular surface of the humeral head. In this case, the accuracy of the X-ray image alignment can be improved by taking advantage of the fact that the distance between that point and the center of the humeral head is equal to the radius of the humeral head approximated by a ball.

[0108] The embodiments of the method relating to this disclosure will be described in more detail below. The humeral head located at the shoulder joint can be approximated by a ball (sphere). Hereafter, unless otherwise specified, the humerus will be approximated by such a ball, which is understood to mean approximating the projection of the humerus in radiographic images by a circle. Therefore, “center” and “radius” will always refer to such an approximated ball or circle. It should also be noted that it is also possible to use other simpler geometric approximations of the humeral head, such as an ellipse. In that case, the anatomical neck will be approximated by an ellipse.

[0109] The following describes an exemplary workflow for determining the entry point. A complex issue in determining the entry point for the humerus is that fractures treated with humeral nails frequently occur along the surgical neck, and therefore the humeral head is displaced. In proper reduction, the center of the humeral head should be close to the humeral diaphysis axis. According to one embodiment, this can be verified with an axial radiograph depicting the proximal humerus. If the center of the humeral head is not close enough to the diaphysis axis, the user is advised to apply traction force distal to the arm to correct the rotation of the humeral head around the articular axis (which may not be detectable). An approximate entry point on the diaphysis axis approximately 20% medial to the center of the humeral head (meaning in the typical axial radiograph above) is then proposed. The user then needs to position an opening instrument (e.g., a k-wire) at this proposed entry point. Alternatively, to improve alignment accuracy as described above, the system requires the user to intentionally position the mouth opener inside the possible entry point (meaning 30–80 percent above the center of the femoral head as depicted in the axial X-ray image) to ensure that the tip of the instrument is definitely within the spherical portion of the humeral head. The system can then detect the humeral head and the tip of this instrument in the new axial X-ray image (for example, by using a neural network).

[0110] Next, the user is instructed to allow only the movement of a certain C-arm (e.g., rotation around the C-axis and further translation) while keeping the tip of the instrument in a fixed position (the tilt of the instrument can be changed) and to acquire an AP image. The humeral head and the tip of the instrument are detected again. Then, the axial image and AP image can be aligned as described above in the paragraph "3D alignment of two or more X-ray images" based on a ball approximating the humeral head and the tip of the instrument.

[0111] The curve that divides the articular surfaces of the shoulder joint is called the anatomical neck. The anatomical neck divides the spherical portion of the humerus, but it is usually impossible for a surgeon to identify it on X-ray. It can be approximated by a 2D circle in 3D space, obtained by the intersection of a ball approximating the humeral head and a plane, and this plane is inclined with respect to the diaphysis axis of the humerus. The spherical articular surface is oriented upward (valgus) and dorsally (when the patient's arm hangs loosely below the shoulder and is parallel to the chest). Three points are sufficient to define this intersection plane. Axial and AP X-rays can each allow for the determination of two points on the anatomical neck, namely the start and end points of the arc that defines the spherical portion of the humerus. Therefore, this is an overdetermination problem, where four points can be determined based on two X-ray images, but only three points are needed to define the intersection plane. If further X-ray images are used, the problem can become even more overdeterministic. This over-determination could allow for more accurate calculations of intersecting surfaces, or it could enable the handling of situations where, for example, a point cannot be determined because it is blocked.

[0112] When determining the approximation of the anatomical neck by having the determined plane intersect with a ball approximating the humeral head, it should be noted that various modifications are possible. For example, the intersecting plane may be shifted laterally to take into account a more accurate position of the anatomical neck on the humeral head. Alternatively, or in addition, the radius of the circle approximating the anatomical neck may be adjusted. It is also possible to use a geometric model with more degrees of freedom to approximate the humeral head and / or the anatomical neck.

[0113] The entry point may be the point on the anatomical neck closest to the intersection of the diaphyseal axis and the bone surface in 3D space, or it may be located at a user-defined distance medially from that point. The anatomical neck and entry point thus determined can be displayed as an overlay on the current X-ray image. If this entry point is very close to the circle approximating the femoral head in the X-ray image, a significant inaccuracy in the z-coordinate may occur. To mitigate this situation, the C-arm may be instructed to rotate so that the proposed entry point moves further medially to the femoral head in the X-ray image. This may be advantageous in any case, as acquiring an X-ray image can be difficult due to mechanical constraints when the entry point is close to the approximation circle. In other words, the rotation of the C-arm between the axial and AP images may be, for example, 60 degrees, which may be easier to achieve in the surgical workflow than a 90-degree rotation.

[0114] Further details, optional implementations, and extensions of this workflow are described in the detailed descriptions of the embodiments below.

[0115] Further methods to enable near real-time continuous 3D alignment of objects This disclosure teaches two further methods that enable determining the imaging direction of an object with a geometric shape (e.g., a small-diameter drill or implant) whose imaging direction cannot be determined without further information, and determining the 3D position and 3D orientation of such an object relative to another object such as a nail, bone, or a combination thereof (i.e., providing 3D alignment of these objects). The first method does not require 2D-3D matching of the object (e.g., a drill), and it may be sufficient to simply detect the location of this object (e.g., the tip of the drill) in two X-ray images. For example, 2D-3D matching of a drill may require stopping the drill and pulling back the sleeve before acquiring an X-ray image, which is cumbersome and prone to errors, and this may be an advantage if a soft tissue protection sleeve is used during drilling. For accurate 2D-3D matching, this pulling back may be necessary even if the drill has already entered the bone, because otherwise the drill bit may not be clearly visible in the X-ray image. The presented method may be advantageous because the drill bit may be rotating and there is no need to pull back the sleeve for acquiring an X-ray image.

[0116] The second method presented here does not require rotation or readjustment of the C-arm (even though changing the C-arm position is not prohibited). For example, in a drilling scenario, this could allow for continuous verification of the actual drill trajectory and comparison of it with the moving space based on X-ray images, while providing near real-time (NRT) feedback to the surgeon at any point during the drilling process.

[0117] In the first method, for example, the 3D position of an identifiable point of an object (e.g., the drill tip) relative to another object (e.g., the sacrum) can be determined by acquiring two X-ray images from different viewing directions (without moving the drill tip between the acquisition of these two images), detecting the drill tip in both X-ray images, aligning them based on one of the procedures presented herein above, and then calculating the midpoint of the shortest line connecting the epipolar lines passing through each drill tip position. The relative 3D orientation of an object (e.g., the drill) can be determined when it is known that the axis of the object contains a specific point whose 3D coordinates relative to the other object (e.g., the sacrum) are known (e.g., the drill axis passes through the entry point on the bone surface, i.e., the position of the drill tip at the start of drilling). When calculating the relative 3D orientation of an object, the potential curvature of the drill and the distortion of the X-ray images in each region may be taken into consideration.

[0118] The second method eliminates ambiguity about the z-coordinate of an object (e.g., a drill) by incorporating prior information that an axis known in the object's coordinate system (e.g., the drill axis) passes through a point whose 3D coordinate relative to another object (e.g., the sacrum) is known (e.g., the entry point, i.e., the starting point of drilling). Furthermore, the potential curvature of the drill and the distortion of the X-ray image in each region can be taken into account in such trajectory calculations.

[0119] If different results are obtained using the first and second methods, this may be due to an inaccurate match of anatomical structures, indicating an inaccurate alignment. This can then be verified by matching the objects in both images. If the match appears to be good, then the matching of anatomical structures, and therefore the image alignment, can be considered good, in which case a mechanical problem is likely. For example, the entry point for drilling into the bone may no longer be on the drill trajectory and should be discarded as a reference point. Then, the currently determined point (e.g., determined by the tip of the drill) can be used as a new reference point for continuing the drilling.

[0120] If the actual drill trajectory does not match the space being worked in (i.e., in the case of distal locking, if the drill continues its current path, it will miss the locking hole for the nail), the system may instruct the user to tilt the power tool by a specified angle while the drill bit is rotating. Doing so will cause the drill bit to laterally cut through the cancellous bone and thus return to the correct trajectory. Such modifications must take into account this additional uncertainty, as this may enlarge the entry hole into the bone and thus shift the position of the original entry point.

[0121] This method also allows for implants consisting of a plate and nail combination with a screw connection between the plate hole and the nail hole. NRT guidance for such implant types proceeds as follows: Based on a 3D reconstruction of the relevant anatomical structure, the ideal position of the combined implant can be calculated by trading off the goodness of the plate position (e.g., surface fit) and the goodness of the nail position (e.g., sufficient distance from the bone surface in a narrow spot). Based on the calculated position, the entry point for the nail into the bone can be calculated. After nail insertion, the ideal position of the combined implant can be recalculated based on the current position of the nail axis. The system can provide guidance to the surgeon to rotate and translate the nail so that the final position of the nail and, where applicable, the sub-implant (e.g., screw), and simultaneously the projected final position of the plate (which is more or less firmly connected to the nail), are optimized. After the nail reaches its final position, the system can provide support for positioning the plate by determining the imaging direction for the plate (which has not yet reached its final target position) on X-ray, taking into account the constraints imposed by the already inserted nail. Next, drilling can be performed through the holes in the plate. This is a crucial step because the drilling must also hit the nail holes, and since re-drilling from a different starting point is not possible, incorrect drilling can be difficult to correct. If the plate is already secured (using screws that do not go through nails), the starting point of the drilling, and therefore the entry point, is also fixed. In such cases, verification and correction of the drill angle can be done multiple times as needed.

[0122] If the holes in the plate only allow drilling at a specific angle, positioning the plate based on the actual nail location can be critical. In such cases, there is no room for further adjustment, and the system can provide guidance for positioning the plate based on the current nail location. This allows the drill trajectory during drilling to be derived simply by aligning the plate with the nail, thereby allowing the drill position to be determined even if only a small portion of the drill is visible on X-ray (the drill tip may still be necessary).

[0123] The proposed system could provide the surgeon with continuous, near real-time guidance. If the alignment is fast enough, a continuous video stream from the C-arm can also be evaluated, providing the surgeon with quasi-continuous navigation guidance. By calculating the relative 3D position and orientation of objects in the current X-ray image and comparing them to the desired coordinates, the surgeon can be given instructions on how to achieve the desired coordinates. Necessary adjustments or movements can be made by the surgeon freehand or supported by mechanical and / or sensory assistance. For example, it is possible to attach an accelerometer to a power tool to support the adjustment of the drill angle. Another possibility is to use a robot that can position one or more of the objects according to the calculated necessary adjustments. Based on the system's NRT feedback, the adjustments can be recalculated at any time and modified as needed.

[0124] It is emphasized that all procedures disclosed herein for addressing situations where the imaging direction to an object cannot be determined are, of course, also applicable to situations where the imaging direction to an object can be determined. Further information obtained from knowledge of the imaging direction can be used to improve accuracy and precision.

[0125] Support for reduction Another objective of the present invention may be to support the anatomically correct reduction of bone fragments. Typically, surgeons attempt to return fractured bone fragments to their original positions in the most natural relative configuration possible. For improved outcomes, it is sometimes important to check whether such reduction is anatomically correct before or after the insertion of fixation implants.

[0126] Reduction can be supported by calculating a 3D reconstruction of the bone of interest. Such a 3D reconstruction does not need to completely reconstruct the entire bone and does not need to be accurate in every aspect. If only specific measures are extracted, the 3D reconstruction needs to be accurate enough to allow for a sufficiently accurate determination of that measure. For example, when determining the femoral anterograde angle (AV), a sufficiently accurate 3D reconstruction of the femur in the condylar and cervical regions may suffice. Other examples of measures of interest may include leg length, degree of leg deformity, curvature (such as femoral lordosis), or the craniocervicidal stipe (CCD) angle, as varus rotation of the proximal femoral fragment often occurs before or after intramedullary nail insertion. Once the measures of interest are determined, they can be used to select an appropriate implant or compared to desired values ​​that may be derived from a database or may be patient-specific (e.g., by comparing the leg to be operated on with the other healthy leg). Instructions on how to achieve the desired values, such as the desired anterograde angle, may be given to the surgeon.

[0127] It might also be interesting to monitor specific measurements throughout the surgery by automatically calculating them from available X-ray images, and perhaps alert the surgeon or trigger appropriate action by the robot if the measurements deviate significantly from the desired values.

[0128] In some cases, 3D reconstruction is possible even from a single radiographic image, especially if the viewing direction can be determined (e.g., based on LU100907B1 or as described herein). However, generally, two or more radiographic images taken from different viewing directions and / or depicting different parts of the bone can increase the accuracy of 3D reconstruction (see the paragraph "Calculation of 3D Representation / Reconstruction" above). Even if a part of the bone is not visible or only partially visible in the radiographic image, a 3D reconstruction can be calculated if the part that is not visible due to a fracture is not displaced relative to the visible part, or if such displacement exists, if the displacement parameter is already known or can be determined by other means. For example, based on a statistical 3D model of the femur, the femoral head can be reconstructed fairly accurately from a pair of ML and AP images where most of the femoral head is not visible. As another example, if the femoral shaft is not fractured, the distal part of the femur can be reconstructed based on two proximal radiographic images. Of course, if further radiographic images showing the distal part are also available, the accuracy of the distal reconstruction may be increased.

[0129] In 3D reconstruction of bone based on two or more X-ray images, accuracy can be further increased if these X-ray images can be aligned before calculating the 3D reconstruction, following one of the techniques described in the paragraph "3D Alignment of Two or More X-ray Images" above. When calculating a 3D reconstruction of bone based on two or more X-ray images showing different parts of the bone (for example, two X-ray images showing the proximal part of the femur and one X-ray image showing the distal part of this femur), 3D alignment of the X-ray images depicting different parts may be possible by using an object with a known 3D model visible in at least one X-ray image for each bone part (e.g., an already embedded nail) and / or by constraining the allowable C-arm movement between the acquisition of these X-ray images.

[0130] The AV angle needs to be determined before or after creating an opening in the patient, before the implant has been inserted (for example, to detect the dorsal gap during reduction of a trochanteric fracture). In such cases, the alignment of two or more images of the proximal femur (e.g., AP and ML) proceeds as follows, following the guidelines in the paragraph "3D alignment of two or more X-ray images" above: When determining the entry point for nail insertion, an opening instrument (of known diameter), such as a k-wire, is placed at the possible entry point and can therefore be detected in the X-ray image. Based on the position of its tip and the detected femoral head, the images can be aligned. If no further objects, such as a k-wire, are visible in the X-ray image, image alignment can still be performed by requesting a specific movement of the C-arm between images. For example, the system may need to rotate 75 degrees around the C-axis of the C-arm. If this rotation is performed with sufficient precision, image alignment can also be performed with sufficient precision. Non-overlapping bone portions (for example, the distal and proximal portions of the femur) can be aligned by restricting the allowed C-arm movement to translation only, as described in one embodiment.

[0131] It should be noted that 3D reconstruction is not essential to determine the AV angle. For example, determining one additional point near the cervical axis may provide sufficient information to determine the AV angle based on 2D methods. Using the above method, it is possible to align 2D structures detected in X-ray images (e.g., structures within the proximal and distal parts of the femur).

[0132] In other cases, for example, when determining the correct rotation angle of a bone, it may be beneficial to consider adjacent bones or bone structures. For example, in the case of a tibial fracture, the femur, patella, and / or fibula condyles may be considered when evaluating the proximal position. Similar comments apply to the evaluation of the distal rotation position. The relative position of the tibia with the fibula or other bone structures (e.g., the overlapping edges of the ankle joint) can clearly indicate the viewing direction to the distal tibia. All these evaluations may be based on a neural network that can perform co-optimization, possibly based on the confidence value (of proper detection) of each structure considered. The results of such evaluations can be combined with knowledge of the patient's or limb's position to assess the current reduction of the bone. For example, in the case of the humerus, the system may instruct the surgeon to position the patient's radius parallel to the patient's body. To evaluate the reduction, it may be sufficient to simply guide the user so that the humeral articular surface is centered relative to the glenoid fossa by detecting these structures in the X-ray image.

[0133] Reduction of X-ray dose It should be kept in mind that the overall objective is to reduce X-ray exposure for patients and operating room staff. The number of X-ray images generated during fracture treatment according to the embodiments disclosed herein should be kept to a minimum. For example, images acquired to check the position of the proximal fragment relative to the distal fragment can also be used to determine the entry point. Alternatively, images generated in the process of determining the entry point can also be used to measure the AV angle or CCD angle.

[0134] According to one embodiment, since it is not necessary to see the complete anatomical structure in the X-ray image, X-ray exposure can also be reduced. Even if objects such as anatomical structures, implants, surgical instruments, and / or parts of implant systems are not visible or only partially visible in the X-ray image, a 3D representation or determination of the imaging direction of the object can be provided. For example, even if the projected image does not fully depict the femoral head, it may still be possible to fully reconstruct the femoral head. As another example, even if the distal portion is not fully depicted, it may be possible to reconstruct the distal portion of the femur based on one or more proximal images.

[0135] In some cases, it may be necessary to determine points of interest associated with anatomical structures, such as the center of the femoral head or specific points on the femoral shaft. In such cases, the points of interest do not necessarily need to be shown in the X-ray image. This is especially true if uncertainty or inaccuracy in determining such points of interest affects dimensions or degrees of freedom that are less important in subsequent development. For example, the center point of the femoral head and / or specific points on the axis of the femoral shaft may be outside the X-ray image, but inaccuracies in the direction of the embedding curve may not significantly affect the calculated embedding curve, so a system, for example based on a deep neural network method, can still determine those points and use them to calculate the embedding curve with sufficient accuracy.

[0136] According to one embodiment, the system's processing unit may be configured to determine anatomical structures and / or points of interest associated with anatomical structures based on an X-ray projection image showing a specific minimum required percentage (e.g., 20%) of the anatomical structure. If less than the minimum required portion of the anatomical structure is visible (e.g., less than 20%), the system may guide the user to obtain the desired view. For example, if the femoral head is not visible at all, the system may instruct the C-arm to move in a direction calculated based on the appearance of the femoral shaft in the current X-ray projection image.

[0137] Matching 3D models to 2D projected images It should be noted that the image data of the processed X-ray images can be received directly from the imaging device, for example, a C-arm, G-arm, or a two-plane 2D X-ray device, or alternatively, from a database. A two-plane 2D X-ray device may have two X-ray sources and receivers offset by any angle. The X-ray projection image may represent anatomical structures of interest, particularly bones. Bones may be, for example, bones of the hand or foot, but more specifically, long bones of the lower limbs such as the femur and tibia, and long bones of the upper limbs such as the humerus, or the sacrum, ilium, or vertebrae. The image may also include bone implants or surgical instruments, such as artificial objects like drills or k-wires.

[0138] In this disclosure, we distinguish between “object” and “model.” The term “object” is used for actual objects, such as bone or a part of bone or another anatomical structure, or implants such as intramedullary nails, bone plates, or bone screws, or surgical instruments such as sleeves or k-wires. An “object” may represent only a part of an actual object (e.g., a part of bone), or it may be a collection of actual objects, and therefore composed of sub-objects (e.g., the object “bone” may be fractured and therefore composed of the sub-object “part of fractured bone”). The term “model” has already been defined above.

[0139] Since 3D representations are essentially computer datasets, it is easy to extract specific information from that data, such as the geometric characteristics and / or dimensions of the virtually represented objects (e.g., axes, contours, curvature, center points, angles, distances, or radii). If the scale is determined based on an object, for example, if the width of a nail is known from model data, then the geometric characteristics or dimensions of another potentially unknown object being depicted can also be measured if such an object is at a similar depth of view. If the depth of view of an object is known (e.g., because the object is large enough or because the size of the X-ray detector and the distance between the image plane and the focal point are known), and there is information about the difference in depth of view between two objects (e.g., based on anatomical knowledge), then the sizes of different objects at different depths of view can also be calculated based on the intercept theorem.

[0140] According to one embodiment, objects in an X-ray image are automatically classified and identified in the X-ray projection image. However, objects can also be manually classified and / or identified in the X-ray projection image. Such classification or identification may be supported by the device by automatically referencing structures recognized by the device.

[0141] Matching an object model to its projection depicted in an X-ray image may consider only selected features of the projection (e.g., contours or distinctive edges) or the entire appearance. Contours or distinctive edges may be determined using a neural segmentation network. The appearance of an object in an X-ray image depends, among other things, on the attenuation, absorption, and deflection of X-ray radiation, which depend on the material of the object. For example, a steel nail generally absorbs more X-ray radiation than a titanium nail, which not only affects the appearance of the projected image of the nail within its contour but can also change the shape of the contour itself, for example, the contour of the nail hole. The strength of this effect also depends on the intensity of the X-rays and the amount of surrounding tissue the X-ray beam must pass through. As another example, transitions between soft and hard tissues may be discernible in X-ray images because such transitions create edges between darker and brighter regions in the image. For example, the transition between muscle tissue and bone tissue may be a recognizable structure, but the internal cortex, which is the transition between spongy internal bone tissue and hard cortical external bone tissue, may also be recognizable as a feature in an X-ray image. It should also be noted that when a bone contour is determined in this disclosure, such a contour may also be the internal cortex or any other recognizable feature of the bone's shape.

[0142] In one embodiment, for objects described by a deterministic model, 2D-3D matching is carried out in accordance with the approach described by Lavallee S., Szeliski R., Brunie L. (1993) Matching 3-D smooth surfaces with their 2-D projections using 3-D distance maps, in Laugier C. (eds): Geometric Reasoning for Perception and Action. GRPA 1991, Lecture Notes in Computer Science, vol. 708. Springer, Berlin, Heidelberg. This method can account for further effects such as image distortion (e.g., pillow effect introduced by an image intensifier) ​​or nail bending by introducing further degrees of freedom into the parameter vector or by using a appropriately tuned model.

[0143] In one embodiment, for an object described by a statistical shape or appearance model, the matching of the virtual projection with the actual projection is carried out in accordance with the approach outlined in V. Blanz, T. Vetter (2003), Face Recognition Based on Fitting a 3D Morphable Model, IEEE Transactions on Pattern Analysis and Machine Intelligence. In this paper, a statistically deformable 3D model is fitted to a 2D image. Therefore, statistical model parameters for contours and appearances, and camera and pose parameters for perspective projection are determined. Another method may follow X. Dong and G. Zheng, Automatic Extraction of Proximal Femur Contours from Calibrated X-Ray Images Using 3D Statistical Models, in T. Dohi et al. (Eds.), Lecture Notes in Computer Science, 2008. By deforming the 3D model so that its virtual projection matches the actual projection of the object in the X-ray image, it becomes possible to calculate the imaging direction (indicating the direction in which the X-ray beam passes through the object).

[0144] When displaying X-ray images, geometric aspects and / or dimensions may be shown as an overlay on the projected image. Alternatively or in addition, at least a portion of the model may be shown on the projected image, for example, as a transparent visualization or 3D rendering, which may facilitate the identification of the structural aspects of the model and thus the imaged object by the user.

[0145] General comments Refer to Figure 25 for the definitions of the rotation and translation axes of the C-arm. In this figure, the X-ray source is indicated by XR, the rotation axis indicated by the letter B is called the vertical axis, the rotation axis indicated by the letter D is called the propeller axis, and the rotation axis indicated by the letter E is called the C axis. Note that in some C-arm models, axis E may be closer to axis B. The intersection of axis D and the central X-ray beam (denoted as XB) is called the "C" center of the C-arm. The C-arm can move up and down along the direction indicated by the letter A. The C-arm can also move along the direction indicated by the letter C. The distance of the vertical axis from the "C" center of the C-arm may vary from C-arm to C-arm. Note that it may also be possible to use a G-arm instead of a C-arm.

[0146] A neural network can be trained on a large amount of data equivalent to the data being applied. When evaluating bone structure in images, the neural network should be trained on a large number of X-ray images of the bone of interest. It will also be understood that a neural network can be trained on simulated X-ray images.

[0147] According to one embodiment, more than one neural network may be used, and each neural network may be specifically trained for the substeps necessary to achieve the desired solution. For example, a first neural network may be trained to evaluate X-ray image data to classify anatomical structures in a 2D projection image, while a second neural network may be trained to detect characteristic edges of that structure in the 2D projection image. A third network may be trained to determine specific key points, such as the center of the femoral head. The neural network may also be combined with other algorithms, including but not limited to model-based algorithms such as active shape models. It should be noted that the neural network may also directly solve one of the tasks of the present invention, for example, determining embedding curves.

[0148] It should be noted that a processing unit can be implemented by a single processor that executes all steps of the process, or by a group or multiple processors that do not need to be located in the same place. For example, cloud computing allows processors to be located anywhere. For instance, a processing unit could be divided into a first subprocessor that controls user interaction, including a monitor for visualizing results, and a second subprocessor (which may be located elsewhere) that performs all calculations. The first or other subprocessor could also control, for example, the movement of the C-arm or G-arm of an X-ray imaging device.

[0149] According to one embodiment, the device may further include storage means that provide a database for storing, for example, X-ray images. It will be understood that such storage means may also be provided in a network to which the system may be connected, and that data related to the neural network may be received via the network. Furthermore, the device may include an imaging unit for generating at least one 2D X-ray image, the imaging unit being capable of generating images from different directions.

[0150] According to one embodiment, the system comprises a device for providing information to a user, the information comprising at least one of a group consisting of X-ray images and instructions relating to procedural steps. It will be understood that such a device may be a monitor or augmented reality device for visualizing the information, or a loudspeaker for providing the information acoustically. The device may further comprise input means for manually determining or selecting the position or part of an object in an X-ray image, such as the contour of a bone, for example, to measure distance in the image. Such input means may be, for example, a computer keyboard, a computer mouse, or a touchscreen for controlling a pointing device, such as a cursor on a monitor screen that may be included in the device. The device may also comprise a camera or scanner for reading package labels or identifying implants or surgical instruments. The camera may also enable the user to communicate visually with the device by gesture or simulation, for example, by virtually touching the device displayed in virtual reality. The device may also comprise a microphone and / or loudspeaker to communicate acoustically with the user.

[0151] It should be noted that all references to C-arm movement in this disclosure always refer to the relative repositioning of the C-arm and the patient. Therefore, translation or rotation of the C-arm can generally be replaced by the corresponding translation or rotation of the patient / operating table, or a combination of C-arm translation / rotation and patient / operating table translation / rotation. This may be particularly relevant when dealing with limbs, as in practice it may be easier to move the patient's limbs than to move the C-arm. It should be noted that required patient movement generally differs from C-arm movement, and usually does not require patient translation, especially if the target structure is already in the desired position on the X-ray image. The system may calculate adjustments to the C-arm and / or patient. It should be further noted that all references to the C-arm also apply to the G-arm.

[0152] The methods and techniques disclosed herein may be used in systems supporting a human user or surgeon, or in systems in which some or all steps are performed by a robot. Therefore, all references to “user” or “surgeon” in this patent application may refer not only to a human user, but also to a robotic surgeon, a mechanical support device, or similar apparatus. Similarly, where it is stated that instructions are given on how to adjust the C-arm, it should be understood that such adjustments may also be performed without human intervention, i.e., automatically by the robotic C-arm, robotic table, or by an OR staff with some automated support. It should be noted that because robotic surgeons and / or robotic C-arms can perform surgery with greater precision than humans, repetitive procedures may require fewer repetitions, and more complex instructions (e.g., combining multiple repetitive steps) may be performed. A key difference between a robotic surgeon and a human surgeon is that the robot can keep the tool completely stationary between acquiring two X-ray images. Where it is required in this disclosure that the tool not move between X-ray images, this may be performed by the robot, or alternatively, the tool may already be slightly fixed within an anatomical structure.

[0153] The computer program may preferably be loaded into the random access memory of a data processing device. Therefore, a data processing device or processing unit in a system according to one embodiment may be equipped to perform at least a portion of the described process. Furthermore, the present invention relates to a computer-readable medium, such as a CD-ROM, on which the disclosed computer program may be stored. However, the computer program may also be provided via a network, such as the World Wide Web, and can be downloaded from such a network into the random access memory of a data processing device. Furthermore, the computer program may also be executed on a cloud-based processor, and the results may be presented via the network.

[0154] Please note that preliminary information about the implant (e.g., nail size and type) may be available before or during surgery by simply scanning the information on the implant packaging (e.g., barcode) or on the implant itself.

[0155] As is evident from the above description, the main aspect of the present invention is the processing of X-ray image data to enable the automatic interpretation of visible objects. The method described herein should be understood as a method to assist in the surgical treatment of patients. Accordingly, according to one embodiment, the method may not include the step of surgical treatment of animals or humans.

[0156] It will be understood that the steps of the method described herein, in particular the steps of the method described in relation to the workflow of embodiments some of which are visualized in the drawings, are major steps, and these major steps may be distinguished or divided into several substeps. Furthermore, there may be further substeps between these major steps. It will also be understood that only a portion of the whole method may constitute the present invention, i.e., steps may be omitted or combined.

[0157] It should be noted that the embodiments are described with reference to different subject matter. In particular, some embodiments are described with reference to method-type claims (computer programs), while others are described with reference to apparatus-type claims (systems / devices). However, from the above and below descriptions, it will be apparent to those skilled in the art that, unless otherwise specified, any combination of features belonging to one type of subject matter, as well as any combination of features relating to different subject matter, are disclosed in this application.

[0158] The embodiments, further embodiments, features, and advantages of the present invention as defined above can also be derived from the examples of embodiments described below, and will be explained with reference to the examples of embodiments shown in the drawings, but the present invention is not limited thereto. [Brief explanation of the drawing]

[0159] [Figure 1] This figure shows a lateral X-ray image of the femur used to determine the entry point for the intramedullary nail. [Figure 2] This figure shows ML X-ray images of the proximal tibia and a mouth opener. [Figure 3] This figure shows AP X-ray images of the proximal tibia and the mouth opener. [Figure 4] This figure shows AP X-ray images of the proximal tibia and the mouth opener. [Figure 5] This figure shows AP X-ray images of the proximal tibia and the mouth opener. [Figure 6] This figure shows the image alignment of the tibia based on two AP X-ray images and one ML X-ray image. [Figure 7] This figure shows an axial X-ray image of the proximal part of the humerus. [Figure 8] This figure shows axial X-ray images of the proximal humerus and the guide rod. [Figure 9] This figure shows AP X-ray images of the proximal humerus and the guide rod. [Figure 10] This figure shows the image alignment of the humerus based on AP X-ray images and axial X-ray images. [Figure 11] This figure shows axial X-ray images of the proximal part of the humerus, a 2D point on the anatomical neck, and a guide rod. [Figure 12] This figure shows AP X-ray images of the proximal part of the humerus, a 2D point on the anatomical neck, and a guide rod. [Figure 13] This figure shows AP X-ray images of the proximal humerus, the 2D projected anatomical neck, the entry point, and the guide rod. [Figure 14]This figure shows AP X-ray images of the proximal humerus, the 2D projected anatomical neck, the entry point, and the guide rod with its tip located above the entry point. [Figure 15] This diagram shows a fractured 3D humerus and guide rod from the AP view direction. [Figure 16] This diagram shows a fractured 3D humerus and guide rod from an axial view. [Figure 17] This diagram shows a fractured 3D humerus and an inserted guide rod from the AP view direction. [Figure 18] This figure shows axial X-ray images of the proximal part of the humerus, a 2D point on the anatomical neck, and the inserted guide rod. [Figure 19] This figure shows AP X-ray images of the proximal humerus, a 2D point on the anatomical neck, and the inserted guide rod. [Figure 20] This figure shows the proximal portion of the femur, its contour, and AP X-ray images of the mouth opener. [Figure 21] This figure shows the proximal portion of the femur, its contour, and ML X-ray images of the mouth opener. [Figure 22] This figure shows an ML X-ray image of the distal femur. [Figure 23] This figure shows ML X-ray images of the distal portion of the femur and its contour. [Figure 24] This diagram shows the 3D femur and the definition of the femoral anterior tilt angle. [Figure 25] This diagram shows the C-arm, its axis of rotation, and its axis of translation. [Figure 26] This figure shows a possible workflow for determining the entry point of the tibia. [Figure 27] This diagram shows a possible workflow for determining the entry point of the humerus. [Figure 28] This figure shows AP X-ray images of the distal femur, the inserted implant, and the drill positioned on the surface of the femur. [Figure 29] This figure shows ML X-ray images of the distal femur, the inserted implant, and the drill positioned on the surface of the femur. [Figure 30] This figure shows the image alignment of the distal femur based on AP and ML X-ray images, including the femur, the inserted implant, and the drill. [Figure 31] This figure shows the same coordinates as Figure 30, but from a different viewing direction. [Figure 32] This figure shows an ML X-ray image of the distal femur with calculated entry points for multiple nail holes. [Figure 33] This figure shows a possible workflow for determining the entry point for an intramedullary implant into the femur. [Figure 34] This figure shows a possible workflow for determining the anterior tilt angle of the femur. [Figure 35] This diagram shows possible workflows for a freehand locking procedure (quick version). [Figure 36] This diagram shows possible workflows for a freehand locking procedure (extended version). [Figure 37] This figure shows a possible workflow for verifying and correcting drill trajectories. [Figure 38] This diagram shows three different drill positions in 3D space. [Figure 39] This figure shows the 2D projection of the scenario in Figure 38. [Figure 40] This figure shows three exemplary workflow examples of methods for supporting autonomous robotic surgery. [Modes for carrying out the invention]

[0160] Throughout the drawings, unless otherwise specified, the same reference numerals and letters are used to indicate similar features, elements, components, or parts of the illustrated embodiments. Furthermore, this disclosure is described in detail herein with reference to the drawings, but this is done in relation to exemplary embodiments and is not limited to the specific embodiments shown in the drawings.

[0161] Methods to support autonomous robotic surgery A first object of the present invention may be to provide a method suitable for supporting autonomous robotic surgery. Firstly, this relates to determining the spatial position and orientation of an object (e.g., a drill, chisel, bone mill, or implant) relative to a moving space related to an anatomical structure. Next, an object of the present invention may be to guide or constrain the movement of the object within the moving space. The system can also control the movement of the object itself. Secondly, this relates to automatically determining when a new alignment (determination of relative spatial position and orientation) based on a new X-ray image is required.

[0162] The movement space can be defined by a 1D subspace such as a line, trajectory, or curve; a 2D subspace such as a plane or a distorted plane; or a 3D subspace in the form of a partial 3D volume. Such subspaces may be restricted (e.g., a line of finite length) or partially unrestricted (e.g., a half-plane). The system may be configured to allow only the movement of an object within the movement space (e.g., by restricting the movement of a robotic arm operated by a surgeon). The system may also be configured to stop the drill when the object leaves the movement space. Alternatively, in a system with a maneuverable arm, an increase in resistance level may be introduced as the object approaches the boundary of the movement space. Multiple movement spaces may exist, each associated with a different system action or response. For example, there may be a first movement space where drilling can take place and a second movement space where the drill can move (without drilling).

[0163] The migration space may be automatically determined by the system based on a model of the anatomical structure, or it may be determined in advance by, for example, a surgeon. The migration space may also exist outside of the anatomical structure and outside of soft tissue. If determined before surgery, it may be re-verified during surgery, possibly incorporating feedback from sensors, such as pressure sensors or cameras.

[0164] Throughout this disclosure, we reiterate that the term “model” should be understood in a very general sense. A model of an anatomical structure may be raw CT data (i.e., 3D image data) of the anatomical structure. A model may also be a processed form of CT data, for example, including segmentation of the surface of the anatomical structure. A model may also be a high-level 3D description of the 3D shape of the anatomical structure, for example, including a description of the surface of the anatomical structure and / or the bone density distribution of the anatomical structure.

[0165] When determining the spatial orientation and position of an object, the system may take into account information provided by several sources, including but not limited to the following: • Sensors that may or may not be attached to the robot (e.g., pressure sensors that measure pressure during drilling) • Information provided by another navigation system (e.g., a navigation system based on one or more cameras, reference bodies and / or trackers and 2D or 3D cameras, a navigation system using infrared, a navigation system with electromagnetic tracking, a navigation system using LiDAR, or a navigation system including wearable tracking elements such as augmented reality glasses) • Information provided by any other intraoperative 3D imaging device (e.g., O-arm) • When a two-plane C-arm is used, X-ray images are acquired by both receivers. • Information from previously acquired X-ray images • Information about the object's previous position • Information about the object's predicted position (this may be based on the object's previous position and how far it has moved; the latter information may be obtainable from the robot itself). • Previously performed system calibration

[0166] Autonomous self-calibrating robots may need to autonomously determine whether and when additional registration procedures are necessary to safely continue a surgical procedure. These additional registration procedures may be based on the acquisition of further X-ray images. This can be triggered by several events or circumstances, including but not limited to: • Input from a sensor (e.g., a pressure sensor indicating that the drill has encountered resistance exceeding a threshold) • Input from an external navigation system that uses sensors to observe surgical procedures (e.g., information that the patient has moved). • Input from an external navigation system indicating that tracking (e.g., forecast) has been interrupted or is currently interrupted. • The alignment performed by the algorithm is not sufficiently accurate (for example, the accuracy of 2D-3D matching of objects in X-ray images falls below the threshold). • The determined position and / or orientation of an object in an image does not match its predicted position (for example, a nail is already embedded in a long bone, and the position of the nail determined by the algorithm does not match the predicted position of the nail). • Certain steps in surgical procedures require particularly high precision (for example, drilling into a particularly dangerous area close to the spinal nerves). • A certain amount of time has passed since the last alignment procedure. • The object has moved a specific distance relative to the anatomical structure (for example, it has been drilled a specific distance). There are reasonable grounds to believe that the 3D scenario may have changed significantly from the previous alignment procedure (e.g., drilling beyond a certain distance, or sensor input indicating patient movement).

[0167] Relative 3D position and 3D orientation can be re-verified with a new alignment procedure. Since this disclosure teaches a near real-time alignment method, any additional alignment procedure performed will incur negligible costs in terms of OR time. The new alignment procedure can be initiated by readjusting the C-arm to acquire new X-ray images from the current C-arm view direction and / or from different view directions. More than one X-ray image may be acquired for increased accuracy. Information provided by an external navigation system may also be taken into consideration. Furthermore, information about the expected position of objects visible in the X-ray (e.g., implants or drill bits) may be incorporated.

[0168] The system can provide instructions to OR staff who require new X-ray images, which may include how to readjust the C-arm. A truly autonomous system can operate the C-arm and / or initiate X-ray image acquisition on its own.

[0169] If a preoperative CT scan is available, the system can perform automated anatomical segmentation and determine the imaging direction to the segmentation in at least one intraoperative X-ray image. Based on prior knowledge of the relative position of the segmentation and the geometric configuration of the object (e.g., a drill), the system determines the imaging direction to the object in the same image, and thus obtains the spatial relationship between the anatomical structure and the object, and based on this, the system can provide instructions or perform actions (i.e., positioning the drill tip and aligning the drill angle). An exemplary workflow with further details is given in Workflow 1 below.

[0170] Alternatively, a workflow without anatomical structure segmentation is also possible. Here, the system can perform image registration by matching preoperative CT scans with intraoperative X-ray images (including registration between the CT scan and all images), or determine individual virtual imaging directions to the (unsegmented) CT scan for each X-ray image. The system can calculate digitally reconstructed radiographs (DRRs) from various imaging directions based on the registered CT scans, and the DRRs can also be registered. Optionally, the system can jointly apply a statistical model of anatomical structure to all available intraoperative X-ray images and DRRs, thereby simultaneously determining the imaging direction to the anatomical structure in all images, which may only be necessary if there is no predetermined movement space defined by the CT scan. Based on this, the system can provide instructions or perform actions as described above. Further details, including how this method is combined with a given anatomical structure segmentation and / or intraoperative CT scan, can be seen in exemplary workflow 2.

[0171] If neither preoperative CT scans nor anatomical segmentation are available, the system can perform image registration and simultaneously fit a statistical model of anatomical structures to the X-ray images. This includes determining the imaging direction to the anatomical structures in all images. Based on this, the system can provide instructions or perform actions as described above. Further details, including how this method can be combined with a given anatomical segmentation and / or intraoperative CT scan, can be seen in exemplary workflow 3.

[0172] Here, consider a situation where the 3D position and orientation of an object relative to the moving space are determined based on two X-ray images from different imaging directions, then a surgical step (e.g., drilling) is performed, and then the new 3D position and orientation of the object relative to the moving space are determined without moving the imaging device. Such a determination may take into account that the possible movements of the anatomical structure relative to the imaging device may be constrained. For example, only the possible translation of the anatomical structure due to the pressure applied to it by the drill may be taken into account.

[0173] 3D reconstruction of anatomical structures can be performed by combining 2D segmented DRRs and 2D segmented real-world X-ray images, with the real-world X-ray images being aligned with the 3D image dataset.

[0174] Three exemplary workflows are provided below. Other implementations may be possible. The step numbers for the three workflows refer to Figure 40.

[0175] Workflow 1: Preoperative CT scan available, with anatomical structural segmentation (see Figure 40)

[0176] 1.1 The system consists of a (deformable) statistical model and a. Direct 3D segmentation of all X-ray images using a neural network, and / or b. Multiple rendered 2D images (DRR) from different known directions using CT scans (i.e., with known image alignment) and image-by-image anatomical structure segmentation using a neural network. Based on this, automated anatomical segmentation of preoperative CT scans is performed.

[0177] 1.2 Based on anatomical structural segmentation and a given screw diameter, the system determines the drill trajectory (i.e., the travel space). If an intraoperative CT scan (or any other 3D scan) is taken during surgery, the system may perform automatic segmentation of this scan and update the initial anatomical structure segmentation and / or drill trajectory.

[0178] 1.3 Option: If the tip of the drill is not necessarily on the surface of an anatomical structure, intraoperative images can be acquired from different directions (e.g., AP, ML, oblique ML) using a fixed drill. The drill must be visible / detectable in 2D.

[0179] 1.4 Options: The system performs image registration (with six optimization parameters per image and six additional parameters regarding the relationship between anatomical structures and the drill). If the drill tip is on the surface of the anatomical structure or at a known distance from it, the relationship between the anatomical structure and the drill requires only five optimization parameters. The system may perform image registration of all available images each time a new image is added, potentially using previous results as an initial guess. Generally, more images lead to greater accuracy.

[0180] 1.5 The system provides instructions for the oblique ML view. This introduces six further optimization parameters for image alignment.

[0181] 1.6 A new X-ray image is acquired.

[0182] 1.7 The system performs image registration of all valid images (e.g., AP, ML, oblique ML), potentially using the results from step 1.4 as an initial guess. Thus, the system determines the segmentation in the latest image and the relative 3D position and 3D orientation with respect to the drill.

[0183] 1.8 The system gives instructions for the drill to move.

[0184] 1.9 Follow the instructions for moving the system drill.

[0185] 1.10 A new X-ray image is acquired.

[0186] 1.11 Based on the predicted and actual drill poses, the system detects whether the C-arm has moved. If C-arm movement is detected, a new image (oblique ML) is acquired, and the workflow returns to step 1.7. Alternatively, the system may take the detected movement into account and proceed to step 1.12.

[0187] 1.12 The system detects whether an anatomical structure has moved relative to the C-arm. The anatomical structure may have moved, for example, due to the sliding of the drill tip and / or a specific pressure (detected by a pressure sensor) exerted by the drill tip. a. If the movement of the anatomical structure exceeds a threshold, the system fits the anatomical structure segmentation to the current image based on previous fittings (six optimization parameters) and determines the relative 3D position and 3D pose of the segmentation and the drill (six optimization parameters, potentially using a fine-tuning step that uses the predicted pose as the initial guess). b. If the displacement of the anatomical structure is below a threshold, the system fits the anatomical structure segmentation to the current image based on previous fittings (e.g., two optimization parameters for the (potentially strabismic) shift of the anatomical structure) and determines the relative 3D position and 3D orientation of the segmentation and the drill (similar to step 1.12a). c. If no anatomical changes are detected, the system determines the segmentation and the relative 3D position and 3D orientation of the drill (similar to step 1.12a).

[0188] 1.13 If the system has not yet given the instruction to start drilling, the system checks whether the drill pose is sufficient. a. If not sufficient, the system returns to step 1.8. b. If sufficient, the system will give a drilling start signal (potentially just a few millimeters of drilling) and return to step 1.9.

[0189] 1.14 Based on the improved entry point and, if available, knowledge of the robot's movement, the drill pose is refined. The system checks whether the drill pose is sufficient. a. If not sufficient, the system returns to step 1.8. b. If sufficient, the system checks whether the planned position has been reached. If not, the system gives an instruction to continue drilling (e.g., a few millimeters). The system returns to step 1.9.

[0190] Workflow 2: Preoperative CT scan available, no anatomical structural segmentation (see Figure 40)

[0191] 2.1 Option: If the drill tip is not necessarily on the surface of an anatomical structure, intraoperative radiographs are taken from different directions (e.g., AP, ML, oblique ML) using a fixed drill. The drill must be visible / detectable in 2D.

[0192] 2.2 Options: The system performs image registration (with six optimization parameters per image and six additional parameters related to the transformation matrix between the preoperative CT scan and the drill). If the drill tip is on the surface of an anatomical structure or at a known distance from it, the drill requires only five optimization parameters. Since segmentation is unavailable, the cost function can be some similarity index between the acquired X-ray image and a specific digitally reconstructed radiograph (DRR) image obtained by rendering the CT scan containing the drill. In addition or alternatively, the system may approximately determine the imaging direction to the drill, for example by a contour-based method, such that the cost function is a weighted average of image similarity and contour similarity. The system may perform image registration of all available images each time a new image is added, potentially using the previous result as the initial guess. Generally, more images lead to greater accuracy.

[0193] 2.3 The system provides instructions for the oblique ML view. This introduces six further optimization parameters for image alignment.

[0194] 2.4 A new X-ray image is obtained.

[0195] 2.5 The system performs image alignment of all available images (including DRR), potentially using the results from step 2.2 as an initial guess. If an intraoperative CT scan (or any other 3D scan) is acquired during surgery, the system can use this scan to perform image registration as well, thus improving accuracy.

[0196] 2.6 If a preoperative surgical plan is available (for example, based on preoperative CT image data), the system uses this information to determine the movement space. The system can further apply a statistical model of the anatomical structure to the aligned images (i.e., actual intraoperative X-ray images and optionally further DRR from, for example, step 2.2) based on contour-based or DRR-based methods. The tip of the drill can be used as a reference point on the surface of the anatomical structure. This reconstruction of the anatomical structure can then be used to provide a more precise movement space. If an intraoperative CT scan (or any other 3D scan) is acquired during surgery, the system can perform automated segmentation of the scan (as described in step 1.1 of workflow 1). In addition or alternatively, the system can use this 3D image data to verify and / or update the determined movement space. If segmentation of preoperative and / or intraoperative CT scans is available, fitting a statistical model may involve using the segmentation as an initial estimate and / or fine-tuning the segmentation. If the movement space has not yet been determined in step 2.6, the system determines the drill trajectory based on the reconstruction of the anatomical structure and a given screw diameter.

[0197] 2.7 Proceed to step 1.8 from workflow 1.

[0198] Workflow 3: Preoperative CT scan is unavailable (see Figure 40)

[0199] 3.1 Option: If the drill tip is not necessarily on the surface of an anatomical structure, intraoperative images are taken from different directions (e.g., AP, ML, oblique ML) using a fixed drill. The drill must be visible / detectable in 2D.

[0200] 3.2 Options: The system performs image registration and anatomical structure reconstruction simultaneously (with six optimization parameters per image, six additional parameters regarding the relationship between anatomical structures and the drill, and additional parameters regarding the deformation of the statistical model). If the tip of the drill is on the surface of the anatomical structure or at a known distance from it, the relationship between the anatomical structure and the drill requires only five optimization parameters. The system may perform image registration and anatomical structure reconstruction for all available images each time a new image is added, potentially using previous results as the initial guess. Generally, more images lead to greater accuracy.

[0201] 3.3 The system provides instructions for the oblique ML view. This introduces six further optimization parameters for image alignment.

[0202] 3.4 A new image is acquired.

[0203] 3.5 The system potentially uses the results from step 3.2 as an initial guess to simultaneously perform image registration of all available images and reconstruction of anatomical structures. The tip of the drill can be used as a reference point on the surface of the anatomical structure. If an intraoperative CT scan (or any other 3D scan) is acquired during surgery, the system can perform automatic segmentation of this scan (as described in step 1.1 of workflow 1). In addition or alternatively, the system can perform image registration based on this scan, and thus accuracy can be improved, for example, by using this image registration as an initial guess. If segmentation from preoperative and / or intraoperative CT scans is available, reconstruction of anatomical structures may include using the segmentation as an initial estimate and / or fine-tuning the segmentation. For example, the system can align the reconstruction of anatomical structures with the automated segmentation, and then fine-tune the initial reconstruction of anatomical structures based on the appearance of the anatomical structures seen in the aligned images.

[0204] 3.6 The system determines the drill trajectory based on the reconstruction of the anatomical structure and a given screw diameter.

[0205] 3.7 Proceed to step 1.8 from workflow 1.

[0206] Comments on all three workflows Statistical models for the reconstruction of anatomical structures may include statistical shape models and statistical appearance models.

[0207] Actions in the workflow can be performed by a surgeon, OR staff, or surgical robot. In the case of a robot, after following system instructions, such as a drill instruction, the robot may confirm its movement. This information may be used by the system to predict, for example, the position of the drill in the next image. If the surgical robot does not provide feedback about its movement, the system cannot predict the position of the drill in the next image. In that case, detection of C-arm movement is skipped (see step 1.11 of workflow 1).

[0208] If, in step 1.11 of workflow 1, you can verify that the C-arm was not moving (for example, the robot was controlling the C-arm), and the robot was providing feedback on its movement, then instead of detecting the C-arm's movement, you could use the difference between the predicted drill pose and the actual drill pose to calibrate the robot (i.e., the robot drill's movement).

[0209] Instead of automatically determining the drill trajectory based on the reconstruction of the anatomical structure (see step 1.2 of workflow 1), the drill trajectory can be determined from a manual preoperative plan. If it is not possible to work with a fixed drill while acquiring images for image registration, the tip of the fixed drill is used as a reference point. This introduces more optimization parameters because the relationship between the anatomical structure and the drill needs to be estimated for each image.

[0210] If two X-ray images depict a portion (always the same portion) of an object that is already inserted into an anatomical structure (for example, a pedicle screw that is already inserted into the pedicle), then image alignment can take this into account.

[0211] If the system knows the precise location of an object (for example, from a previous image), this information can be used for subsequent image alignment.

[0212] Determining the entry point for implanting an intramedullary nail in the femur. Another objective of the present invention may be the determination of the entry point and implantation curve for implanting an intramedullary nail in the femur. To determine the entry point, it is necessary to acquire X-ray images from a specific viewing direction. In a true lateral view, the diaphyseal axis and cervical axis are parallel with a specific offset. However, this view is not the desired view of the present invention. The desired view is a lateral view rotated around the C-axis of the C-arm so that the implantation axis passes through the center of the femoral head. The center of the femoral head can be determined with sufficiently high accuracy, for example, by a neural network. The uncertainty in determining the center of the femoral head may mainly relate to deviations in the direction of the implantation axis, which does not significantly affect the accuracy that guarantees the desired viewing direction. The system can support the user in obtaining the desired viewing direction by estimating the required rotation angle around the C-axis based on an anatomical structure database or LU100907B1.

[0213] The system can also help the user obtain the correct viewing orientation. For example, consider a scenario where the 2D distance between the center of the femoral head and the tip of the mouth opener is too small compared to the 2D distance between the tip of the mouth opener and the lowest visible part of the femoral shaft. This effect occurs when the focal axis of the C-arm is nearly perpendicular to the implantation axis. In such a case, the center of the shaft at its narrowest point would likely not be visible in the current X-ray projection image. Therefore, the system may instruct the user to rotate the C-arm around axis B in Figure 25. Following this instruction results in an X-ray projection image in which the first distance increases and the second distance decreases (i.e., the cervical region becomes larger and the narrowest part of the shaft becomes visible).

[0214] As mentioned above, one way to determine the angle at which the C-arm needs to be rotated to obtain the desired view is to consider the anatomical appearance in the AP radiograph. The following points, namely the center of the femoral head, the tip of the mouth opener, and the center of the diaphysis at the transition to the greater trochanter, can be identified in the image. Two lines can then be drawn between the first two points and the last two points, respectively. Since these three points can be identified with sufficient accuracy in the ML radiograph as well, it may be possible to estimate the angle between the focal line in the ML radiograph and the anatomical structure (e.g., the implantation axis and / or the cervical axis). If this angle is too small or too large, the system can provide instructions to increase or decrease the angle, respectively.

[0215] According to one embodiment, the implantation axis may be determined as follows. Figure 1 shows a lateral (ML) X-ray image of the femur. The system may detect the center of the diaphysis (denoted as ISC) and the center of the femoral head (denoted as CF) at the narrowest point. The line defined by these two points can be considered the implantation axis (denoted as IA). Furthermore, the system may detect the projected lateral boundary (denoted as OB) of the cervical and diaphysis regions, or alternatively, multiple points on the boundary. For example, a neural network may perform boundary segmentation. Alternatively, the neural network may directly estimate specific points rather than the complete boundary. For example, the neural network may estimate the center of the diaphysis rather than the boundary of the diaphysis, and estimate the diameter of the diaphysis based on the size of the femoral head. Based on this information, it may be possible to estimate the location of the diaphysis boundary without finding the boundary itself. The implantation axis should be at a specific distance from both the cervical boundary and the diaphysis boundary. If either distance is too small, the system can calculate the necessary rotation of the C-arm around the C-axis to achieve the desired viewing direction in the subsequently acquired X-ray projection image. The direction of rotation of the C-arm may be determined based on a weighted assessment of the distance in the cervical region and the distance in the diaphyseal region. The rotation angle may be calculated based on an anatomical model of the femur.

[0216] Upon reaching the desired viewing direction, the intersection of the implantation axis and the trochanteric periphery axis may be defined as the entry point. The trochanteric periphery axis can be detected directly in the image. If this is undesirable or impractical, the trochanteric periphery axis can also be approximated in the X-ray image by a line connecting the tip of the aperture device to the implantation axis. This line may be considered perpendicular to the implantation axis, or it may extend at an oblique angle to the implantation axis if prior information, if available, suggests otherwise.

[0217] The implant may consist of a nail and a head element. If the distance between the projected tip of the mouth opener and the projected entry point is not within the desired distance (e.g., the distance is greater than 1 mm), the system may guide the user on how to move the mouth opener to reach the entry point. For example, if the tip of the mouth opener on the femur is positioned excessively forward compared to the determined entry point, the system will instruct the user to move the tip of the mouth opener backward.

[0218] According to one embodiment, the system can detect the narrowest part of the femoral shaft, the center of the femoral head (referred to as CF), and the tip of the mouth opener (referred to as KW) using X-rays. The implantation axis (referred to as IA) can be considered as a line passing through the center of the femoral head (referred to as CF) and the center at the narrowest part of the shaft (referred to as ISC). The entry point can be considered as the point on the implantation axis closest to the tip of the mouth opener KW (referred to as EP). The system can be instructed to move the mouth opener to position it on the EP. After moving the instrument to the projected location, acquiring AP images may be useful to verify in the AP view that the tip of the mouth opener is still at the projected tip of the greater trochanter. It's possible that, based on the alignment of AP and ML images, knowledge of the epipolar rays projected from the tip of the k-wire detected in the AP image exists, and if there is no movement of the k-wire tip between the acquisition of the AP image and the acquisition of the ML image, this could lead to a more accurate determination of the entry point, potentially eliminating the need to further verify with another AP image whether the k-wire tip is still at the projected tip of the greater trochanter.

[0219] An example of a possible workflow for determining the entry point for an intramedullary implant into the femur (see Figure 33):

[0220] 1. The user obtains an AP X-ray image with the tip of the mouth opener positioned at the projected tip of the greater trochanter.

[0221] 2. The user acquires an ML X-ray projection image without moving the tip of the mouth opener.

[0222] 3. The system detects the center of the femoral head, the center point of the narrowest part of the diaphysis, and the tip of the mouth opener using X-ray images. a. If both the femoral head and the narrowest part of the diaphysis are not clearly visible, the system instructs the C-arm to move laterally to broaden the field of view. b. If only the femoral head is not fully visible, but the narrowest part is fully visible, the system instructs the C-arm to move proximal along the leg. c. The system calculates a first distance between the center of the femoral head and the tip of the mouth opener, and a second distance between the tip of the mouth opener and a specific point on the diaphysis. This point may be the center of the diaphysis at its narrowest point (if visible), or, if the narrowest point is not visible, the visible distal point of the diaphysis, or alternatively, the estimated center of the diaphysis at its narrowest point (based on the visible portion of the diaphysis). d. If only the diaphysis is not fully visible, but the femoral head is fully visible, the system instructs the C-arm to move distally along the leg. One way to determine if the diaphysis is fully visible may be to compare the second distance from step 3c to a threshold. Another way may be to assess the curvature of the diaphysis to determine if the narrowest part is visible in the current X-ray image. e. If the first distance from step 3c is too small compared to the second distance, the C-arm must be rotated clockwise (right femur) or counterclockwise (left femur) around the C-arm axis B (see Figure 25), and vice versa. The angle by which the C-arm needs to be rotated can be calculated based on the two distances and possibly further information from the AP image from step 1. The latter may include, for example, the CCD angle of the femur. The curvature of the diaphysis as depicted in the ML X-ray image may also be taken into consideration.

[0223] 4. Steps 2 and 3 are repeated until all important parts of the femur are clearly visible and the two distances from step 3c are in the desired ratio.

[0224] 5. In addition to the points from step 3, the system detects the left and right outlines of the femoral neck and the left and right contours of the femoral shaft.

[0225] 6. A line is drawn from the center of the femoral head to the center of the narrowest part of the femoral shaft. Four distances are calculated between this line and the four contours of the femoral neck and femoral shaft.

[0226] 7. For both the cervical and diaphyseal regions, a metric is defined to evaluate how close the line passes to the center of each region. For example, the cervical metric is 0 when the line touches the left contour of the neck, 1 when the line touches the right contour of the femur, and 0.5 when the line is at the center of the cervical region.

[0227] 8. A new metric is defined based on the weighted average of the cervical and diaphyseal metrics. If the new metric is lower than the first threshold, the C-arm must be rotated around its C-axis so that the focal point of the C-arm moves forward. If the new metric exceeds a second threshold that is higher than the first threshold, the C-arm must be rotated in the opposite direction around its C-axis. The angle by which the C-arm must be rotated can be calculated based on the distance between the metric and the corresponding threshold.

[0228] 9. If the metric defined in step 8 falls outside the two thresholds from step 8, a new ML X-ray projection image must be obtained.

[0229] 10. Steps 5-9 are repeated until the metric defined in step 8 falls between the two thresholds from step 8. The drawn line is the final projected embedding axis.

[0230] 11. The distance between the projected tip of the opening device and the line from step 10 is calculated.

[0231] 12. Option: The tip of the access instrument is detected. Based on the appearance of the tip of the access instrument (i.e., its size in the X-ray projection image), the system gives an instruction to move the tip of the access instrument backward or forward.

[0232] 13. If the tip of the access instrument is too far away from the line in step 10, its position is optimized and a new ML X-ray projection image is acquired.

[0233] 14. Steps 11 - 13 are repeated until the tip of the access instrument is within a specific distance from the line in step 10.

[0234] 15. An AP X-ray projection image is acquired to confirm that the tip of the access instrument is still at the tip of the greater trochanter. If not, return to step 2.

[0235] Procedure for implanting a nail with a sub - implant into the tibia Example of a possible workflow (see Figure 26):

[0236] 0. In the following workflow, it is assumed that the proximal part of the tibia is not damaged (or has been properly repositioned).

[0237] 1. The user places the access instrument on the surface of the tibia (any point on the proximal part, but ideally near the entry point estimated by the surgeon).

[0238] 2. The user acquires a (substantially) lateral image of the proximal part of the tibia (denoted as TIB) as depicted in Figure 2.

[0239] 3. The user acquires at least one AP image of the proximal part of the tibia (ideally, multiple images from slightly different directions) as depicted in Figures 3, 4, and 5.

[0240] 4. The system detects the size (or diameter, etc.) of the opening instrument (denoted as OI) in all images in order to estimate the size (scaling) of the tibia.

[0241] 5. The system jointly matches the statistical model of the tibia to all images, for example, by matching the statistical model to the bone contour (or more generally, the bone appearance). The result of this step is the 3D reconstruction of the tibia. a. This includes six parameters per image regarding rotation and translation, one parameter regarding scaling (already initially estimated in step 4), and a specific number of modes (the determination of the modes corresponds to the 3D reconstruction of the tibia). Thus, if there are n images and m modes, the total number of parameters is (6·n + 1 + m). b. Based on all the estimated rotations and translations of the tibia (in each image), the system performs image registration of all images as depicted in FIG. 6. Thus, the spatial relationships between the AP images (denoted as I.AP1 and I.AP2), the ML image (denoted as I.ML), the tip of the opening instrument (denoted as OI), and the tibia (denoted as TIB) are known. c. Option: For potentially more accurate results, the system may use information of the femur head or fibula, for example, by using its statistical information.

[0242] 6. Based on the 3D reconstruction of the tibia, the system determines the entry point. This can be done, for example, by defining the entry point on the average shape of the statistical model. Then, this point can be identified in the 3D reconstruction.

[0243] 7. Option: Based on the 3D reconstruction of the tibia, the system (virtually) places an implant on the bone and calculates the length of the proximal locking screw. This step can also improve the estimation of the entry point as it takes into account the actual implant.

[0244] 8. The system displays the entry point as an overlay on the current X-ray image.

[0245] 9. If the tip of the opening device is not close enough to the estimated entry point, the system will instruct the user to adjust the position of the tip. a. The user adjusts the position of the tip of the mouth opener and obtains a new X-ray image. b. The system calculates the entry point in the new image (for example, by image difference analysis or by matching a 3D reconstruction of the tibia to the new image). c. Return to step 8.

[0246] 10. The user inserts an implant into the tibia and obtains a new image.

[0247] 11. The system determines the imaging direction for the implant. Based on the 3D reconstruction of the tibia, the system provides the necessary 3D information (e.g., the length of the proximal locking screw).

[0248] 12. The system provides support for proximal locking.

[0249] 13. The system calculates the torsional angle by comparing the proximal tibia (which may include the femoral condyle) with the distal tibia (which may include the foot). To calculate the torsional angle more accurately, the system may also use information about the fibula.

[0250] Procedure for implanting a sub-implant with a nail into the humerus Examples of possible workflows (see Figure 27):

[0251] 0. The user provides a desired distance (e.g., 0 mm or 5 mm medial) between the entry point and the anatomical neck.

[0252] 1. The user obtains an axial X-ray image of the proximal humerus, as depicted in Figure 7.

[0253] 2. The system detects the contour of the humeral head (e.g., using a neural network). Based on the detected contour, the system approximates the humeral head (denoted as HH) with a circle, i.e., estimates its 2D center and radius. This may include multiple candidates for the humeral head (2D center and radius), which are ranked based on their validity (e.g., based on a statistical model, mean squared approximation error, confidence level, etc.). Based on the detected diaphysis axis (denoted as IC), the system rotates the image so that the diaphysis axis is a vertical line. The system evaluates whether the center of the humeral head is close enough to the diaphysis axis. If the distance between the center of the humeral head and the diaphysis axis is too large, the system advises the user to apply a distal tensile force to the arm to correct the redisplacement by translation (i.e., humeral head versus diaphysis, where the force from the soft tissue results in a redisplacement perpendicular to the tensile force).

[0254] 3. The system estimates the initial entry point (denoted as EP) located somewhere between the intersection of the humeral head and the diaphysis axis (for example, 20% above the center of the intersection).

[0255] 4. The user positions the guide rod at the initial estimated entry point from step 3.

[0256] 5. The user obtains further axial X-ray images in which the guide rod (labeled OI) is visible, as depicted in Figure 8.

[0257] 6. The system detects the humeral head (denoted as HH) (2D center and radius) and the 2D diaphysis axis (denoted as IC), and the tip of the guide rod (denoted as OI) and its 2D scaling (based on the known diameter of the guide rod).

[0258] 7. The system advises the user to rotate the C-arm around its C-axis (furthermore, permissible C-arm movements are distal-proximal or anterior-posterior translations, while prohibited movements are other rotations and inward-outward translations).

[0259] 8. As shown in FIG. 9, the user obtains an AP X-ray image of the proximal part of the humerus (not necessarily a true AP image) without moving the tip of the guide rod (angular movement of the guide rod is allowed as long as the tip remains in a fixed position).

[0260] 9. The system detects the humeral head (denoted as HH) (2D center and radius) and the 2D shaft axis (denoted as IC), and detects the tip of the guide rod (denoted as OI) and its 2D scaling (based on the known diameter of the guide rod).

[0261] 10. Based on the information from Steps 6 - 9, the system performs image registration as shown in FIG. 10 and calculates a spherical approximation of the humeral head (denoted as HH3D) and a 3D shaft axis in the same coordinate system as this sphere.

[0262] 11. There are four points (denoted as CA in FIGS. 11 and 12), two for the axial image and two for the AP image, that define the start and end points of the circular portion of the projected humeral head. The system detects at least three of these four points. Based on these at least three points, the system determines the 3D anatomic neck (for example, by defining a plane based on three points that intersect the spherical approximation of the humeral head).

[0263] 12. The system may also use the fourth point from Step 11 (for example, in a weighted least squares method where the weights are based on the individual confidence levels of the four points) to improve the determination of the anatomic neck.

[0264] 13. When the anatomical structure is virtually rotated in space such that the 3D shaft axis becomes vertical and the humeral head is above the shaft, the entry point is defined as the highest point in the space on the anatomic neck (denoted as CA3D in FIG. 13). Based on the settings from Step 0 and the results from Steps 10 - 12, the system calculates the final entry point (denoted as EP).

[0265] 14. The user positions the guide rod at the calculated entry point and acquires a new AP X-ray image as depicted in Figure 14.

[0266] 15. The system detects the tip of the guide rod (referred to as OI) and evaluates whether the tip of the guide rod is sufficiently close to the calculated entry point (referred to as EP).

[0267] 16. Steps 14 and 15 are repeated until the tip of the guide rod is close enough to the entry point.

[0268] 17. Voluntary control over the angular motion of the guide rod. a. Based on the latest image alignment (including 3D humeral head), the system determines the spatial relationship between the humeral head and the guide rod, as depicted in Figures 15 and 16. If the orientation of the guide rod deviates significantly from the intended insertion direction, the system provides instructions for the angular motion of the guide rod. The intended insertion direction can be estimated, for example, using a statistical model or by comparing the axis of the guide rod (denoted as OIA) with the axis of the humeral head (denoted as HA). b. If instructions are given in step a, the user follows the instructions and acquires a new X-ray image from the same direction. Image difference analysis detects changes in the image and updates the image alignment. c. Steps a and b are repeated until no further angular motion of the guide rod is required.

[0269] 18. Voluntary improvement of image alignment and verification of the contour of the humeral head. a. The user inserts the guide rod as depicted in Figure 17. b. The user acquires an axial X-ray image (for example, as depicted in Figure 18). c. The system determines the imaging direction to the guide rod (indicated as OI) and detects the humeral head (indicated as HH) (2D center and radius). d. The system advises the user to rotate the C-arm around its C-axis (see step 7 for further possible C-arm movements). e. The user acquires an X-ray image from another direction (e.g., AP as depicted in Figure 19) without moving the guide rod. f. The system determines the imaging direction to the guide rod (indicated as OI) and detects the humeral head (indicated as HH) (2D center and radius). g. Based on the information from both images, the system performs image alignment. This image alignment is more accurate than in step 10 because the 3D model of the guide rod is known. Based on image alignment, the system can verify the detection of the humeral head in both images. i. Based on the validation results, the system optimizes the contour of the humeral head in both images (for example, by selecting a different candidate for the humeral head).

[0270] 19. Voluntary correction of rotational displacement of the humeral head. a. The user acquires (axial or AP) X-ray images. The system determines the imaging direction to the guide rod and detects the 2D diaphysis axis and the 2D humeral head axis (defined by the visible circular portion of the humeral head). b. If the previous image was acquired in a significantly different orientation (for example, the previous image was axial and the current image is AP), the system performs image registration based on the most recent image pair. Based on this image registration, the system determines the ideal 2D angle between the diaphysis axis and the femoral head axis of the current image. c. If the previous image was taken from a very similar orientation (e.g., identified by image difference analysis), the ideal 2D angle between the diaphysis axis and the femoral head axis remains unchanged (compared to the previous image). d. The system calculates the current 2D angle between the diaphysis axis and the femoral head axis. e. If the angle between the diaphyseal axis and the femoral head axis is not close enough to the ideal angle from step 19b or 19c (e.g., 20° in the axial image or 130° in the AP image), the system provides instructions to correct the rotational displacement in the dorsoventral (axial image) or medial-lateral (AP image) direction. f. If the previous image was acquired from a very similar orientation, but the circular area showing the humeral head is smaller or larger than in the previous image (for example, due to a previous displacement correction), the rotational displacement may have changed in other orientations as well. Therefore, the system gives further instructions to rotate the C-arm around its C-axis to change the orientation of the next image (i.e., update the image alignment). g. If instructed, the user corrects the rotational displacement (by rotating the C-arm if necessary) and returns to step 19a.

[0271] 20. Check for voluntary twisting. a. The user positions their forearms parallel to their body (or thighs). b. The user acquires an axial X-ray image. c. The system detects the humeral head axis and the 2D center of the glenoid fossa. The system calculates the distance between the center of the glenoid fossa and the humeral head axis. Based on this result, the system instructs on which direction and by what angle the twist needs to be corrected. d. The user corrects the twist by rotating the femoral head by an angle in the direction from step C. e. Steps 20b to 20d are repeated until the center of the glenoid fossa is sufficiently close to the axis of the humeral head.

[0272] Potential changes: Instead of estimating the entry point 20% above (inward) from the center of the intersection in step 3, the system may use a higher value (e.g., 70%) to ensure that the tip of the guide rod is on the spherical portion of the humeral head. In step 10, the system may use the information that the tip of the guide rod is on the spherical approximation of the humeral head to improve image registration. With the 70% method described above, the current position of the tip of the guide rod is a greater distance to the entry point (compared to the 20% method). When guiding the user to reach the entry point with the tip of the guide rod (steps 14-16), the system determines whether the view direction has changed (e.g., by image difference analysis). If the view direction has not changed, the entry point calculated from previous X-ray images is used, and the guidance information is updated based on the detected position of the updated tip. If the view direction has changed slightly, the entry point is shifted accordingly (e.g., by a technique called object tracking, see, e.g., SR Balaji et al., “A survey on moving object tracking using image processing” (2017)). If the viewing direction changes significantly, the system instructs the user to rotate the C-arm around its C-axis without moving the tip of the guide rod, and to acquire an X-ray image from a different viewing direction (e.g., axial if the current image is AP). Based on the updated image, the system performs image alignment based on information obtained from previous alignments (e.g., the radius of the ball approximation of the humeral head), displays the entry point on the current image, and navigates the user so that the tip of the guide rod reaches the entry point.

[0273] Determination of the anterior tilt angle of the femur Below is an exemplary workflow for determining the AV angle before or after implant insertion, which may be more robust and / or accurate than the latest technologies. According to one embodiment, the overall procedure for determining the femoral anterior tilt angle is as follows (see Figure 34).

[0274] 1. The user positions the tip of the mouth opener approximately at the tip of the greater trochanter.

[0275] 2. The user obtains an AP X-ray image of the proximal femur, as depicted in Figure 20.

[0276] 3. The system detects the 2D contour of the femur (denoted as FEM), the femoral head (denoted as FH) approximated by a circle (i.e., determined by the 2D center and 2D radius), and the tip of the mouth opener (denoted as OI).

[0277] 4. If several important parts of the femur or the tip of the mouth opener are not sufficiently visible, the system will instruct the user to rotate and / or move the C-arm, and the user will return to step 2.

[0278] 5. The user rotates the C-arm around its C-axis to acquire ML X-ray images. The user may further use inward, outward, and / or forward / backward shifts of the C-arm. The tip of the mouthpiece must not be moved while the C-arm is being moved.

[0279] 6. The user obtains an ML X-ray image of the proximal femur, as depicted in Figure 21.

[0280] 7. The system detects the 2D contour of the femur (represented as FEM) and the femoral head (represented as FH) (i.e., the 2D center and 2D radius), and the tip of the mouth opener (represented as OI).

[0281] 8. If several important parts of the femur or the tip of the mouth opener are not sufficiently visible, the system will instruct the user to move the C-arm (translation only) or rotate the C-arm around its C-axis, and the user will return to step 6.

[0282] 9. Based on the proximal AP and ML image pairs, the system performs image alignment. If image alignment is unsuccessful, the system instructs the C-arm to rotate and / or move, and the user returns to step 2.

[0283] 10. The user moves the C-arm distally along the patient's leg. In this step, rotation of the C-arm is not permitted, but translation in all three directions is allowed.

[0284] 11. The user obtains ML radiographic projection images of the distal femur, as depicted in Figures 22 and 23.

[0285] 12. The system detects the 2D contour (referred to as FEM) of the femur.

[0286] 13. No specific position or alignment of the femoral condyles is required. However, if some important parts of the femur are not clearly visible, the system will instruct the user to move the C-arm (only translation is permitted), and the user will return to step 11.

[0287] 14. Based on image registration, the system jointly fits the statistical model (trained on fractured and unfractured femurs) to all images so that the projected contour of the statistical model matches the detected 2D contour of the femur in all images. This step directly leads to the 3D reconstruction of the femur. To improve the accuracy of the 3D reconstruction, the system can calculate the 3D position of the tip of the mouth opener (based on proximal image registration) and use this point as a reference point, taking advantage of the fact that the tip of the mouth opener is positioned on the surface of the femur.

[0288] 15. The system determines the anteversion angle based on a 3D reconstruction of the femur, as depicted in Figure 24. According to Yeon Soo Lee et al.: “3D femoral neck anteversion measurements based on the posterior femoral plane in ORTHODOC® system” (2006), the anteversion angle can be calculated based on the center of the femoral head (denoted as FHC), the center of the femoral neck (denoted as FNC), the posterior vertex of the trochanter (denoted as TRO), and the lateral and medial vertices of the posterior femoral condyle (denoted as LC and MC). The system identifies these five points in the 3D reconstruction of the femur from step 10 and thus calculates the anteversion angle.

[0289] Freehand locking procedure Various implementations of distal locking procedures for femoral nails are possible. Below are two examples of possible workflows (one "quick" and the other "enhanced" in precision). In either workflow, the user can verify the drill trajectory based on X-ray images with near real-time (NRT) feedback at any point during drilling and adjust the drilling angle as needed. This verification does not require rotation or readjustment of the C-arm. An exemplary workflow for such verification is provided below.

[0290] See Figure 35 for an example of a possible workflow (quick version):

[0291] 1. The user obtains an X-ray image of the distal femur (e.g., AP or ML as depicted in Figure 28).

[0292] 2. The system determines the imaging direction to the implant and detects the contour of the femur. If neither the implant nor the femur contour can be detected, the system instructs the user to improve visibility (for example, by moving the C-arm). The user follows the instructions and returns to step 1.

[0293] 3. The user positions the drill on the surface of the femur (e.g., the nail hole trajectory). The user acquires an X-ray image from a different viewing direction (e.g., 25°-ML as depicted in Figure 29).

[0294] 4. The system determines the imaging direction to the implant (referred to as IM), detects the contour of the femur (referred to as FEM), and determines the relative 3D position and 3D orientation of the implant and the drill (referred to as DR).

[0295] 5. If the drill tip cannot be detected, the system will instruct the user to improve the visibility of the drill tip (for example, by moving the C-arm). The user follows the instructions, obtains a new image, and returns to step 4.

[0296] 6. Based on the determination of the imaging direction of the implant in both images (labeled I.AP and I.ML in Figure 30), the system performs image registration as depicted in Figures 30 and 31.

[0297] 7. Based on the image alignment from step 6, the system fits a statistical model of the femur by matching its projected contour in both images with the detected contour of the femur (i.e., determining the rotation and translation, scaling, and mode of the statistical model of the femur in both images).

[0298] 8. For the current image, the system defines a line from the drill tip to the focal point in the image plane. This line intersects the reconstructed femur twice (i.e., the entry and exit points). The point closer to the focal point is selected as the current 3D position of the drill tip. The system may calculate the length of the locking screw based on the diameter of the femoral diaphysis reconstructed along the nail hole trajectory.

[0299] 9. Based on the known spatial relationship between the femur and the implant (achieved through image registration and femur reconstruction), the system calculates the spatial relationship between the drill and the implant.

[0300] 10. When the drill trajectory passes through the nail hole, the system instructs the user to begin drilling, and the user begins drilling and proceeds to step 14. At any point during the drilling process, the user can verify the drill trajectory according to the following exemplary workflow.

[0301] 11. If the drill trajectory does not pass through the nail hole, the system will instruct the user to move the drill tip and / or rotate the drill. The user will follow the instructions and obtain a new X-ray image.

[0302] 12. The system evaluates whether the view direction has changed (e.g., by image difference analysis). If the view direction has not changed, the system can use most of the results from the previous image to determine the imaging direction to the drill. If the view direction or any other relevant image content has changed (e.g., due to image blurring effects, occlusion, etc.), the system can use this information to improve image alignment (e.g., by using further view directions of the current image). The system determines the imaging direction to the implant and drill, detects the contour of the femur, and fits the reconstructed femur to the current image.

[0303] 13. The user returns to step 9.

[0304] 14. If the user wishes to lock with additional holes, the system displays the entry points for all nail holes (given by the intersection of the 3D reconstruction of the femur and the embedding curve of the ideal lock position) and provides instructions on how to move the drill tip to reach the entry points. An example is depicted in Figure 32. The user positions the drill tip at the calculated entry point (denoted as EP) and returns to step 12.

[0305] For an example of a possible workflow (extended version), see Figure 36:

[0306] 1. Optional: The user acquires an X-ray image of the distal femur (e.g., AP or ML as depicted in Figure 28). The system determines the imaging direction to the implant (labeled IM) and detects the femoral contour (labeled FEM). If neither the implant nor the femoral contour can be detected, the system instructs the user to improve visibility (e.g., by moving the C-arm). The user follows the instructions and returns to the beginning of this step.

[0307] 2. The user positions the drill on the surface of the femur (e.g., the nail hole path).

[0308] 3. The user acquires an X-ray image of the distal femur (e.g., ML or AP). The system determines the imaging direction to the implant (labeled IM), detects the femoral contour (labeled FEM), and determines the relative 3D position and 3D orientation of the implant and drill (labeled DR). If the implant, femoral contour, or drill tip cannot be detected, the system instructs the user to improve visibility (e.g., by moving the C-arm). The user follows the instructions and returns to the beginning of this step. Based on the 3D reconstruction of the bone relative to the nail coordinate system, the system calculates the required length of the sub-implant (e.g., locking screw) and displays the corresponding information.

[0309] 4. The user acquires an X-ray image from a different viewing direction (e.g., 25°-ML as depicted in Figure 29). The drill tip must not move between images. If it has moved, the system can detect this and may prompt the user to return to step 3.

[0310] 5. The system determines the imaging direction to the implant (referred to as IM), detects the contour of the femur (referred to as FEM), and determines the relative 3D position and 3D orientation of the implant and the drill (referred to as DR).

[0311] 6. If the drill tip cannot be detected, the system will instruct the user to improve the visibility of the drill tip (for example, by moving the C-arm). The user follows the instructions, obtains a new image, and returns to step 5.

[0312] 7. Based on the determination of the imaging direction of the implant in at least two images (labeled I.AP and I.ML in Figure 30), the system performs image registration as depicted in Figures 30 and 31.

[0313] 8. Based on the image registration from step 7, and possibly using information from previous image registrations, the system fits a statistical model of the femur by matching its projected contour in the image with the detected contour of the femur (i.e., determining the rotation and translation, scaling, and mode of the statistical model of the femur in both images). Optional: The system may update the calculated sub-implant length based on the reconstructed bone and the determined nail hole trajectory.

[0314] 9. For the current image, the system defines a line L1 (labeled L1 in Figure 31) from the tip of the drill to the focal point in the image plane. L1 intersects the reconstructed femur twice (i.e., the entry and exit points). The point closer to the focal point is selected as the initial value for the current 3D position of the drill tip.

[0315] 10. For images from other viewing directions, including the drill tip, the system defines a line L2 from the drill tip to the focal point (i.e., the corresponding coordinate system of the image) in the image plane. Based on image alignment, this line is transformed to the coordinate system of the current image. The transformed line is called L2' (labeled L2' in Figure 31).

[0316] 11. If the minimum distance between L1 and L2' exceeds a certain threshold, the system may advise the user to return to step 4, as this likely indicates that the drill tip has moved between images. Optional: If the user confirms that the drill tip has not moved between the generation of the image pair used for image alignment, the system will improve image alignment by optimizing the determination of the implant's imaging direction in both images and minimizing the distance between L1 and L2'. (If the determination of the imaging direction to the implant and the detection of the drill tip are perfect in both images, and the drill tip has not moved between images, L1 and L2' will intersect.)

[0317] 12. The point on L1 that has the minimum distance to L2' is selected as the first further value of the current 3D position of the drill tip.

[0318] 13. Based on the two solutions for the 3D position of the drill tip (i.e., from steps 9 and 12), the system finds the current 3D position of the drill tip (for example, by selecting the solution from step 12 or by averaging both solutions). Since the drill tip is on the surface of the femur, the system improves the 3D reconstruction of the femur under the constraint that the estimated 3D position of the drill tip is on the surface of the reconstructed femur. Based on the improved reconstruction of the femur, the system may verify the previously calculated sub-implant length. If the updated length deviates from the previously calculated screw length (perhaps taking into account the increment of available length of the sub-implant), the system notifies the user.

[0319] 14. Based on the known spatial relationship between the femur and the implant (achieved through image registration and femur reconstruction), the system calculates the spatial relationship between the drill and the implant.

[0320] 15. When the drill trajectory passes through the nail hole, the system instructs the user to begin drilling, and the user begins drilling, inserts the sub-implant after drilling, and then proceeds to step 19. At any point during the drilling process, the user can verify the drill trajectory according to the following exemplary workflow.

[0321] 16. If the drill trajectory does not pass through the nail hole, the system will instruct the user to move the drill tip and / or rotate the drill. The user will follow the instructions and obtain a new X-ray image.

[0322] 17. The system evaluates whether the view direction has changed (e.g., by image difference analysis). If the view direction has not changed, the system determines the imaging direction to the drill, although it may use most of the results from the previous image. If the view direction or any other relevant image content has changed (e.g., due to image blurring effects, occlusion, etc.), the system may use this information to improve image alignment (e.g., by using further view directions of the current image). The system determines the imaging direction to the implants, optimized by determining the imaging direction of already inserted sub-implants, if available, taking into account available information about their entry points and drills, detects the contour of the femur, and fits the reconstructed femur to the current image.

[0323] 18. The user returns to step 14.

[0324] 19. If the user wishes to lock with additional holes, the system displays the entry points for all nail holes (given by the intersection of the 3D reconstruction of the femur and the embedding curve of the ideal lock position) and provides instructions on how to move the drill tip to reach the entry points. An example is depicted in Figure 32. The user positions the drill tip at the calculated entry point (denoted as EP) and returns to step 17.

[0325] If the user decides at any point to check whether the hole lock was successful, an image can be acquired in an imaging direction that deviates by less than 8 degrees from the locked hole trajectory, and the system will automatically evaluate whether the lock was successful. If the system has information that the last hole is locked or that the performed locking procedure needs to be verified, the system may guide the user to position the C-arm so that it reaches above the locked hole trajectory.

[0326] To support the execution of skin incisions at the correct spot for positioning the drill at the proposed entry point, the system can project the skin entry point based on the embedding curve and the entry point on the bone by estimating the distance between the skin and bone.

[0327] See Figure 37 for an example of a possible workflow for verifying and correcting drill trajectories:

[0328] 1. The user acquires an X-ray image from the current imaging direction.

[0329] 2. The system aligns the drill and the nail; that is, it determines their relative 3D position and orientation based on the acquired X-ray images. The ambiguity of 2D-3D matching can be resolved by taking into account prior information that the drill axis passes through the entry point (i.e., the starting point of drilling) (the 3D coordinates relative to the nail are predetermined in the workflow shown in Figure 35 or Figure 36). Further explanation of this is provided below.

[0330] 3. If the current drill position and orientation relative to the nail indicates that the drill will miss the lock hole if it continues on its current path, the system instructs the user to tilt the power tool by a specified angle while the drill bit is rotating. Doing so will cause the drill bit to laterally cut through the cancellous bone and thus return to the correct trajectory. The angle given in the instruction takes into account that following the instruction may cause the drill to bend inside the bone, and the amount of bending may depend on the drill insertion depth, bone density, and the stiffness and diameter of the drill.

[0331] 4. The user may return to step 1 or resume drilling. This loop from step 1 to step 4 can be executed continuously with near real-time navigation guidance.

[0332] The resolution of the ambiguity in the 2D-3D matching in Step 2 is shown in Figures 38 and 39. Figure 38 shows three different drill positions in 3D space (denoted as DR1, DR2, and DR3), all of which correspond to the same 2D projection DRP in Figure 39. However, by taking into account the prior information that the drill axis passes through the entry point EP, the ambiguity regarding the 3D position and orientation of the drill relative to the nail N can be resolved.

[0333] Note that as the drill approaches the nail, the tip of the drill in the X-ray image may overlap with the nail, potentially preventing the resolution of 2D-3D matching ambiguity in the image acquired in step 1. In this case, a possible solution is to acquire further X-ray images from different imaging directions showing the tip of the drill (and the nail). The imaging direction to the nail can also be determined in the further X-ray images, and therefore the further X-ray images can be aligned with the original X-ray image. The tip of the drill may be detected in the further X-ray images. The point defined by the detected drill tip in the further X-ray images defines the epipolar line. The axis of the tool may be detected in the original X-ray image and define the epipolar plane. The intersection of the epipolar plane and the epipolar line defines the position of the tip relative to the nail in 3D space.

Claims

1. A computer program product that, when executed on a processing unit of a system for autonomous robotic surgery, To perform surgical procedure steps, control the movement of the robotic device. The control of the aforementioned movement is based on information including the spatial position and orientation of at least a portion of the robot device. Based on the trigger information, the system receives a projected image, which is generated when the robot device is not moving. The projected image is processed, and the spatial position and orientation of at least a part of the robot device are determined based on the projected image, and To perform the next surgical procedure step, control the movement of the robotic device. A computer program product configured in such a way.

2. The computer program product according to claim 1, further configured to generate trigger information, the trigger information being generated based on data received from at least one of the group consisting of sensors of a robotic device, a navigation system, a tracking system, a camera, previous projection images, intraoperative 3D scans, and a definition of a moving space.

3. A computer program product according to any one of claims 1 to 2, further configured to determine the deviation of a determined spatial position and orientation from an expected spatial position and orientation, and to generate calibration information based on the determined deviation.

4. The computer program product according to any one of claims 1 to 3, further configured to determine the imaging direction of the next projection image.

5. A computer program product according to any one of claims 1 to 4, further configured to cause an imaging device to generate a projected image.

6. The computer program product according to any one of claims 1 to 5, further configured to control the imaging device to move to a new position for generating a projected image from a different imaging direction.

7. The computer program product according to any one of claims 1 to 6, wherein the control of the movement of the robot device is further based on at least one of the group consisting of further image processing of projected images, information from a tracking system, information from a navigation system, information from a camera, information from a LiDAR, information from a pressure sensor, and calibration information.

8. A system for autonomous robotic surgery, comprising a processing unit configured to execute a computer program product according to any one of claims 1 to 7.

9. The system according to claim 8, further comprising a robotic device, wherein the computer program product is configured to control the movement of the robotic device.

10. The system according to any one of claims 8 to 9, further comprising an imaging device, wherein the computer program product is configured to control the movement of the imaging device and / or the generation of a projected image by the imaging device.

11. The system according to any one of claims 8 to 10, further comprising at least one device from the group consisting of a navigation system, a tracking system, a camera, and a sensor, wherein the control of the movement and / or the determination of the spatial position and orientation are further based on information received from the at least one device.

12. A method for autonomously performing surgical procedure steps using a robotic device, wherein the method is: A step of controlling the movement of a robotic device in order to perform a surgical procedure step, wherein the control of the movement is based on information including the spatial position and orientation of at least a portion of the robotic device, A step of pausing the movement of the robot device based on trigger information, The steps include receiving a projection image of at least a portion of the robot device, The steps include processing the projection image and determining the spatial position and orientation of at least a portion of the robot device based on the projection image, A step of controlling further movement of the robotic device in order to perform the next surgical procedure step, Methods that include...

13. The method according to claim 12, further comprising the step of generating trigger information, wherein the trigger information is generated based on data received from at least one of the group consisting of sensors of a robotic device, a navigation system, a tracking system, a camera, previous projection images, intraoperative 3D scans, and a definition of a moving space.

14. The method according to any one of claims 12 and 13, wherein the control of the movement and / or the determination of the spatial position and orientation are further based on information received from at least one device from the group consisting of a navigation system, a tracking system, a camera, and a sensor.

Images