Simple and Reliable 3D Navigation System and Method for Musculoskeletal Surgery Based on a Single 2D X-ray Image

Abstract

When executed by a processing unit of a system, there is provided a computer program product including instructions for causing the system to receive a single X-ray image, a 3D data set describing a region of interest within a patient's anatomical structure, and a 3D model of a surgical object. The X-ray image is a 2D projection image depicting at least a portion of the surgical object including an anchor point and the region of interest. The computer program product is configured to determine the 2D position of the anchor point in the X-ray image and the imaging direction to the region of interest based on the X-ray image. Based on this information, the 3D position of the anchor point in the 3D coordinate system defined by the 3D data set is determined. Further, based on the 3D model of the surgical object and the 3D position of the anchor point in the 3D coordinate system, the 3D position and 3D orientation of the surgical object relative to the region of interest are determined by the computer program product.

Description

Technical Field

[0001] The present invention relates to the field of computers and robot-assisted surgery. Further, the present invention relates to a system and method for providing information about surgical objects and anatomical structures based on CT data or another type of 3D X-ray scan and X-ray images. In particular, the present invention relates to a system and method for automatically determining the spatial position and orientation of a surgical object relative to a patient. This method can be implemented as a computer program executable on a processing unit of the system.

Background Art

[0002] Spinal surgery, particularly spinal fusion, is a commonly performed surgical procedure. However, spinal surgery also poses significant risks to patients. Incorrect drilling in the spine can damage the spinal cord, cause serious injury to the nerves or membranes of the spinal cord, and may lead to chronic pain, permanent paralysis, incontinence, or sexual dysfunction. To reduce these risks associated with incorrect drilling, computer-assisted navigation is already used at a certain frequency in spinal surgery. Computer assistance is related to the surgeon's navigation to ensure that drilling is performed in the correct location and pedicle screws are properly placed. This includes determining the accurate relative 3D position and 3D orientation between a surgical instrument (such as a drill) and the patient (e.g., relative to a desired drilling trajectory).

[0003] Almost all existing navigation systems require additional procedural steps and devices such as 3D cameras, trackers, and reference bodies. In spinal surgery with navigation, most current systems use optical tracking with a dynamic reference system attached to the spine (for patient tracking) and a reference body attached to the surgical instrument. In that case, both references must always be visible to the 3D camera. Such an approach has several drawbacks including, but not limited to: ·The validity and accuracy of registration must be continuously monitored. If the tracker moves, it may be necessary to repeat the registration. ·Failure to notice the movement of the tracker can result in inaccurate navigation instructions and harm to the patient. ·Accuracy decreases as the distance from the camera increases.

[0004] In summary, existing navigation systems are error-prone. A long chain of errors in navigation workflows such as registration, instrument calibration, and real-time tracking means that the accuracy of the navigation system is only as good as the weakest part at the root of the chain. Therefore, even when working with a navigation system, surgeons need to constantly verify the accuracy of navigation instructions, for example, by tactile and landmark confirmation, which are uncertain verification procedures.

[0005] Robot-assisted surgical systems are becoming popular due to the accuracy they are thought to provide. However, since robots operate substantially based on the instructions provided by navigation systems, the accuracy of robots is only as good as the information provided by these navigation systems. Existing navigation systems are error-prone, so current robotic surgeons cannot be trusted to perform surgical procedures autonomously. Rather, robotic surgeons are merely automated holding arms that hold instruments. The actual drilling still needs to be done by the surgeon.

[0006] The main problem with conventional navigation systems is that they do not determine the actual relative 3D position and orientation between surgical instruments and anatomical structures, but instead infer their position and orientation by tracking with a camera reference bodies externally attached to the tool (e.g., a drill) and the anatomical structure. By its nature, this camera can only see externally fixed reference bodies and cannot see the drill itself inside the bone. If any one of the reference bodies moves, or if the drill bends inside the bone, the navigation system may not notice this and provide inaccurate information, potentially harming the patient.

Summary of the Invention

Problems to be Solved by the Invention

[0007] It is desirable to have a navigation system that (i) does not require additional procedures and devices for navigation, and (ii) can reliably determine the actual 3D position and orientation of a surgical object (e.g., a tool or implant) with respect to the patient. Such a navigation system is particularly useful in spinal surgery, but can also be used in many other surgical procedures where the spatial relationship of a surgical object or implant with respect to the patient must be accurately determined. It can also be used in combination with the aforementioned conventional navigation systems to provide redundant navigation information, thereby making the navigation very robust and enabling autonomous robotic surgery at this level of safety.

Means for Solving the Problems

[0008] At least one of the problems mentioned is alleviated or solved by the subject matter of each of the independent claims. Further embodiments are described in each of the dependent claims.

[0009] A system and method are proposed that do not require a reference body or a tracker to determine the 3D spatial relationship (i.e., relative 3D position and 3D orientation) of a surgical object with respect to a part of a patient's anatomical structure. The proposed system and method (implemented as a computer program product) are configured to determine the 3D spatial relationship based on a single X-ray image depicting at least a part of the surgical object and an area of interest within at least the patient's anatomical structure.

[0010] Generally, when executed by a processing unit of the system, a computer program product is provided that includes instructions for causing the system to receive an X-ray image, a 3D dataset describing at least the area of interest, and a 3D model of the surgical object. The X-ray image is typically a 2D projection image depicting at least a part of the surgical object, which usually includes anchor points, and an area of interest within the patient's anatomical structure. The computer program product is configured to determine the 2D position of the anchor points in the X-ray image and the imaging direction to the area of interest based on the X-ray image. Based on this information, the 3D position of the anchor points in the 3D coordinate system defined by the 3D dataset is determined. Further, based on the 3D model of the surgical object and the 3D position of the anchor points in the 3D coordinate system, the computer program product determines the 3D position and 3D orientation of the surgical object with respect to the area of interest.

[0011] The surgical object may be, for example, a surgical instrument or tool such as a drill, a needle, or a saw, or it may be an implant such as a pedicle screw, a nail, or a plate. The surgical object has a known geometric shape described by the 3D model. The surgical object also has points, hereinafter referred to as anchor points, throughout the present disclosure, and the positions of the anchor points in the 3D model of the surgical object are known and can be identified in the X-ray image. The anchor point can be, for example, the tip of a drill. In the case of a saw, the anchor point can be one edge or the midpoint of the saw blade.

[0012] A 3D data set that describes a part of a patient's anatomical structure can be generated by a computed tomography (CT) scan or some other type of 3D X-ray scan obtained preoperatively or intraoperatively. The 3D data set includes a region of interest, which must be sufficiently rigid. The region of interest can be, for example, a bone fragment that can be considered sufficiently rigid, a particular vertebra, or a volume that includes two adjacent vertebrae. The region of interest can also be the entire 3D data set. Since the 3D data set accurately describes at least the patient's anatomical structure within the region of interest, a one-to-one correspondence is established between points in the physical world and points in the 3D data set, at least within the region of interest. Thus, the 3D data set can define a 3D coordinate system within the physical 3D space. The purpose is to determine the 3D position and orientation of a surgical object in this 3D coordinate system, or in other words, to determine the 3D spatial relationship of the surgical object (in the physical world) with respect to the region of interest within the patient's anatomical structure.

[0013] According to one embodiment, in preoperative or intraoperative planning, target points in the 3D data set and within the region of interest are determined. The target points can be determined preoperatively or intraoperatively by the surgeon manually, automatically by the system, or semi-automatically by the surgeon with some automatic support by the system. Thus, the position of the target points in the 3D coordinate system defined by the 3D data set is known. The target points are selected such that the anchor points of the surgical object are placed at (or near) the physical position of the target points in the patient, which can be achieved by selecting target points on the bone surface.

[0014] In another embodiment, the target point may be the starting point of the target path, and its end point is called the target end point. The target path is within the region of interest, and its position and orientation within the 3D data set can be determined manually by the surgeon, automatically by the system, or semi-automatically by the surgeon with some automatic support by the system in preoperative or intraoperative planning. Thus, when the target path is available, its position and orientation in the 3D coordinate system are known. For example, when treating a vertebra, the target path corresponds to a planned drilling trajectory (usually a straight line), and the target point is located on the surface of the vertebra.

[0015] The methods disclosed herein teach how to determine the 3D position and 3D orientation of a surgical object in a 3D coordinate system defined by a 3D data set that describes the patient's anatomical structure. Thereby, the relative 3D position and 3D orientation of the surgical object with respect to the region of interest of the patient's anatomical structure are determined. Thus, when the region of interest includes a planned target path, the relative 3D position and 3D orientation of the surgical object with respect to the target path in the patient's anatomical structure can also be determined. This can also be, for example, an object that provides an indication for aligning the surgical object with the target path (if available) for drilling along the target path.

[0016] Neither the target point nor the target path is necessarily required. According to one embodiment, it is also possible to work only with anchor points (e.g., the tip of a drill). In this case, the surgeon determines during the operation whether the 3D position and orientation of the surgical object with respect to the patient's anatomical structure are appropriate and, if not, how they should be corrected.

[0017] It is emphasized that it is not necessary to perform segmentation of the 3D dataset. In particular, (other than selecting target points on the bone surface) it is not necessary to identify the bone surface in the 3D dataset. Furthermore, it is not necessary to know, nor explicitly determine, the exact position and shape of the region of interest within the 3D dataset. For example, when treating a vertebra, a CT scan may include not only this vertebra but also several adjacent vertebrae (and surrounding soft tissue). First, it is not necessary to identify the vertebra in the 3D dataset. A rough approximation of the region of interest within the 3D dataset can be derived from the target points. The anchor points define the 2D region of interest in the X-ray image. In subsequent calculations, data points within the 2D region of interest can be weighted, with lower weights applied to points further from the anchor points.

[0018] The imaging direction to the 3D region of interest depicted in the X-ray image can be determined by calculating DRRs (digitally reconstructed radiographs) of multiple imaging directions from the 3D dataset. Alternatively, a model of the structure to be imaged may be utilized to determine the imaging direction. The imaging direction may also be determined based on position sensors and / or movement sensors provided on the imaging device.

[0019] Throughout this disclosure, the term "imaging direction" (also referred to as "view direction") to an object (or region of interest) means the 3D angle through which the X-ray beam passes through a selected point of the object (or region of interest). In other words, the imaging direction to the region of interest represents the orientation of the X-ray device with respect to the anatomical structures within the region of interest.

[0020] When adopting a DRR that best matches an X-ray image to determine the imaging direction, a numerical optimizer can also be used to find the optimal DRR. It is not necessary to constrain the DRR to the region of interest. When evaluating the best match between the DRR and the X-ray image, the region of interest may be emphasized (e.g., with appropriate weighting). If the acquired X-ray image is distorted (i.e., acquired with an image intensifier rather than a flat panel receiver), since the greater the distance between points, the greater the potential for distortion, it may be necessary to use a smaller region within the X-ray image for DRR matching (or emphasize points closer to the anchor point). Since distortion usually occurs more strongly in regions further from the central X-ray beam, it may be beneficial to ensure that the anchor point is reliably near the center of the X-ray image.

[0021] As detailed in US2021 / 0248779A1, the 3D pose (i.e., 3D position and orientation) of a thin surgical object such as a drill cannot always be uniquely determined based on a single X-ray image. Thus, generally, when only one X-ray image is available, there may be ambiguity in the relative 3D spatial relationship between the surgical object and the patient's anatomical structure. The present disclosure teaches resolving such ambiguity by first determining the 3D position of the anchor point in the 3D coordinate system defined by the 3D data set. Various ways to achieve this are disclosed.

[0022] In a method according to an embodiment that does not use target points, the anchor point needs to be placed on the bone surface. Based on a single X-ray image, the 2D position of the anchor point in the X-ray image is determined, and from the DRR matching, the imaging direction of the X-ray image to the region of interest can be determined. In 3D space, a line is determined by a point that is located on the anchor point of the virtual projection onto the X-ray image. Since the anchor point is on the bone surface, the position of the anchor point in 3D can be found by determining points on the line located on the bone surface. It is sufficient to simply determine the position of the bone surface along the line. It is not necessary to segment the entire surface of the bone.

[0023] In another method, a correspondence between an anchor point (on the surgical object) and a target point (in the 3D data set) is established. This is done, for example, by virtually projecting the target point onto the X-ray image using a known imaging direction to the region of interest determined through, for example, DRR matching.

[0024] If the position of the anchor point in the X-ray image coincides with the position of the target point, the 3D position of the anchor point in physical space can be considered to coincide with the 3D position of the target point. This is because the target point is selected such that the anchor point of the surgical instrument is placed on the target point (in physical space). Of the three coordinates of the 3D position, two degrees of freedom are determined by the X-ray image, and the third degree of freedom is determined by prior information that the target point and the anchor point are on or at a defined distance from the bone surface.

[0025] Typically, a 3D data set (e.g., a CT scan) includes scale information (i.e., information regarding size) of the anatomical structures described in the data set. Furthermore, the scale of the scene depicted in the X-ray image can also be provided by a depicted surgical object of known exact geometric shape, or another surgical object of known geometric shape that may be depicted in the X-ray image (e.g., a previously implanted pedicle screw). This redundancy regarding scale information can be used to distinguish surgical objects by comparing the scale of the 3D data with the size of the surgical objects when different surgical objects in the X-ray image have the same geometric shape but different sizes (e.g., drills of different diameters). If the acquired X-ray image is distorted, the same information (i.e., the 3D data set, the 3D model of the surgical object, or another surgical object of known geometric shape) can also be used to correct the distortion.

[0026] If the position of the anchor point in the X-ray image does not coincide with the position of the target point (virtually projected), the system provides an indication of how to move the surgical instrument so that the anchor point approaches the target point, and then a new X-ray image can be acquired. If the position of the anchor point in the X-ray image is in the vicinity of the target point and a local model of the bone surface in the vicinity of the target point is available, the 3D position of the anchor point can be obtained from the local model for the 2D position of the anchor point detected in the X-ray image. This is sufficient to determine the 3D relative spatial relationship between the surgical object and the patient's anatomical structure. Such a local model of the bone surface can be, for example, the median shape of the bone, a first approximation of the shape of the bone, or a local segmentation of the bone surface in the vicinity of the target point. The appropriate model is selected based on the labeling of the 3D dataset (e.g., classification of vertebrae), which can be done automatically, semi-automatically, or manually. The size of the vicinity of the target point can be adapted based on the local model. For example, if there is less variation in the local model closer to the target point, a larger vicinity can be selected. Of course, if a segmentation of the 3D dataset (i.e., a model of the entire bone surface) is available, taking this into account can improve the accuracy.

[0027] It is again emphasized that it is not necessary to know the exact position and shape of the 3D region of interest within the 3D dataset. Furthermore, it is not necessary to establish an exact correspondence between the 2D region of interest in the X-ray image and the 3D region of interest in the 3D dataset. However, there may be several possible 3D regions of interest. For example, in spinal surgery, since multiple vertebrae need to be treated, there may be multiple 3D regions of interest. In such cases, a human surgeon manually identifies which 3D region of interest to treat. It is also possible to automatically select the relevant 3D region of interest corresponding to the anchor point indicated by the surgical instrument by performing DRR matching for each possible 3D region of interest and selecting such a best match.

[0028] The region of interest is an approximation of a region (or volume) that can be considered rigid enough for the 3D dataset to describe the patient's anatomical structure within the region of interest with sufficient accuracy. For example, since the spine is flexible, individual vertebrae can move relative to each other. If the patient moves after acquisition of the 3D dataset, some relative movement of the individual vertebrae is expected. This means that, assuming no large movement due to breathing, the region that can be considered rigid may be larger for intraoperative 3D x-rays (with no subsequent patient movement) than for preoperative CT scans (with subsequent patient movement).

[0029] The methods taught by this disclosure may also supplement existing navigation techniques. Another aspect may be to continuously incorporate information from intraoperative x-rays. The systems and methods disclosed herein do not require a navigation camera or other sensors, but nevertheless may be combined with a navigation camera or other sensors (e.g., sensors attached to a robot) (to increase accuracy and / or redundancy). Combining information from x-ray images with information from a camera or other sensors can improve the accuracy of determining relative spatial positions and orientations, or resolve remaining ambiguities (which may result from occlusion, for example). If the tool is held by a robot or robotic arm, information provided by the robot or robotic arm itself (e.g., about the movements made) may also be considered.

[0030] Possible applications for applying this disclosure include any type of bone drilling, such as insertion of screws into the pedicle, osteotomy, insertion of screws into the sacroiliac joint, insertion of screws connecting two vertebrae, insertion of screws through the pelvis, insertion of screws through the acetabulum, or drilling for the cruciate ligament. The teachings disclosed are used for drilling, reaming, milling, chiseling, sawing, excision, and positioning of implants and can thus support, for example, osteotomy, tumor resection, total hip arthroplasty.

[0031] It will be understood that the system includes a processing unit and the method can be implemented as a computer program product executable on that processing unit.

[0032] According to one embodiment, the system and method can cause an imaging device to generate an X-ray image or some type of 3D X-ray scan (e.g., a CT scan). Further or alternatively, the system and method can control the imaging device to move to a new position to generate X-ray images from different imaging directions. Such different imaging directions can be appropriate imaging directions proposed by the system. The current imaging direction can be determined based on an internal movement sensor and / or a position sensor of the imaging device.

[0033] Note that the image data of the processed X-ray image can be received directly from the imaging device, e.g., a C-arm, a G-arm, or a two-plane 2D X-ray device, or alternatively from a database. The two-plane 2D X-ray device has two X-ray sources and receivers offset by any angle.

[0034] According to one embodiment, objects in the X-ray image can be automatically classified and identified in the X-ray projection image. However, the objects can also be manually classified and / or identified in the X-ray projection image. Such classification or identification can be supported by the device by automatically referring to the structures recognized by the device.

[0035] Note that the processing unit can be implemented by only one processor that executes all steps of the process, or by a group of or multiple processors that do not have to be located in the same place. For example, with cloud computing, processors can be located anywhere. For example, the processing unit can be divided into a first sub-processor that controls the interaction with the user including a monitor for visualizing the results, and a second sub-processor (which may be located in a different place) that executes all calculations. The first sub-processor or another sub-processor can also control, for example, the movement of the C-arm or G-arm of an X-ray imaging device.

[0036] According to one embodiment, the device may further comprise storage means for providing, for example, a database for storing X-ray images. Such storage means may also be provided in a network to which the system can be connected, and it will be understood that data related to the neural network can be received via that network. Furthermore, the device comprises an imaging unit for generating at least one 2D X-ray image, and the imaging unit is capable of generating images from different directions.

[0037] According to one embodiment, the system comprises a device for providing information to a user, the information including at least one piece of information from the group consisting of X-ray images and instructions regarding steps of a procedure. Such a device may be a monitor or an augmented reality device for visualizing the information, or it may be understood that it may be a loudspeaker for providing the information acoustically. The device may further comprise input means for manually determining or selecting the position or part of an anatomical structure in an X-ray image, such as the outline of a bone, for example, to measure distances in the image. Such input means may be, for example, a computer keyboard, a computer mouse, or a touch screen for controlling a pointing device, such as a cursor on a monitor screen that may be included in the device. The device may also comprise a camera or a scanner for reading a label of a package or identifying a surgical object. The camera may also enable the user to communicate visually with the device by means of gestures or simulations, for example, by virtually touching a device displayed in virtual reality. The device may also comprise a microphone and / or a loudspeaker for communicating acoustically with the user.

[0038] It should be noted that all references in this disclosure to the movement of the C-arm always refer to the relative repositioning of the C-arm and the patient. Thus, a translational or rotational movement of the C-arm can generally be replaced by a corresponding translational or rotational movement of the patient / operating table, or a combination of a translational / rotational movement of the C-arm and a translational / rotational movement of the patient / operating table. This may be particularly relevant when handling the limbs, as it may sometimes be easier to move the patient's limbs than to move the C-arm. It should be noted that the required movement of the patient generally differs from the movement of the C-arm, and in particular, when the target structure is already in the desired position in the X-ray image, it is usually not necessary to translate the patient. The system may calculate the adjustment of the C-arm and / or the adjustment of the patient. It should be further noted that all references to the C-arm apply equally to G-arms, O-arms, etc.

[0039] The methods and techniques disclosed in the present invention may be used in a system that supports a human user or surgeon, or in a system where some or all steps are performed by a robot. Thus, references to "user" or "surgeon" in this patent disclosure may refer not only to a human user, but also to a robotic surgeon, a mechanical support device, or a similar apparatus. Similarly, when it is described that instructions are given on how to adjust a C-arm, such adjustments may also be performed automatically, i.e., by a robotic C-arm, a robotic table, without human intervention, or may be performed by OR staff with some automated support. Note that since a robotic surgeon and / or a robotic C-arm can perform surgery with higher accuracy than a human, repetitive procedures require fewer repetitions and more complex instructions (e.g., combining multiple repetitive steps) can be executed. The main difference between a robotic surgeon and a human surgeon is that the robot can keep the tool completely stationary while acquiring two X-ray images.

[0040] The computer program product may preferably be loaded into the random access memory of a data processing device. Thus, the data processing device or processing unit of a system according to one embodiment may be equipped to execute at least a part of the described process. Further, the present disclosure relates to a computer-readable medium such as a CD-ROM on which the disclosed computer program product may be stored. However, the computer program product may also be provided via a network such as the World Wide Web and can be downloaded from such a network into the random access memory of a data processing device. Further, the computer program product may also be executed on a cloud-based processor and the results may be presented via a network.

[0041] Note that prior information about surgical objects (e.g., drill size and type) may be obtained before or during surgery by simply scanning a description on a package (e.g., barcode) or on the surgical object itself.

[0042] As is apparent from the above description, the main aspect is to process X-ray image data to enable automatic interpretation of visible objects. The methods described herein should be understood as methods for assisting in the surgical treatment of patients. Thus, according to one embodiment, this method may not include steps of treating an animal or a human body by surgery.

[0043] The steps of the methods described herein, particularly those related to the workflows according to embodiments where some of them are visualized in the drawings, are main steps, and it will be understood that these main steps may be distinguished or divided into several sub-steps. Furthermore, there may be additional sub-steps between these main steps. It will also be understood that only a part of the overall method may constitute the invention, i.e., steps may be omitted or combined.

[0044] It should be noted that embodiments are described with reference to different subjects. In particular, some embodiments are described with reference to method-type claims (computer programs), while other embodiments are described with reference to apparatus-type claims (systems / devices). However, from the above and the following descriptions, it will be apparent to those skilled in the art that any combination of features belonging to one type of subject, as well as any combination between features related to different subjects, are considered to be disclosed in this application, unless otherwise specified.

[0045] The above-defined aspects and further aspects, features, and advantages of the present invention can also be derived from the examples of the embodiments described below and will be described with reference to the examples of the embodiments shown in the drawings, but the present invention is not limited thereto.

Brief Description of the Drawings

[0046]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0047] Throughout the drawings, unless otherwise specified, the same reference numerals and characters are used to indicate similar features, elements, components, or parts of the illustrated embodiments. Further, although the present disclosure is described in detail herein with reference to the drawings, this is done in connection with exemplary embodiments and is not limited to the specific embodiments shown in the drawings.

[0048] The present disclosure teaches a system and method for determining the 3D position and orientation of a surgical object with respect to a portion of a patient's anatomical structure called a region of interest described by a 3D data set such as a CT scan or some other type of 3D X-ray scan based on a single X-ray image. Without additional information, the 3D pose (i.e., 3D position and orientation) of a thin surgical object such as a drill cannot always be uniquely determined based on a single X-ray image.

[0049] FIG. 1 shows a 2D X-ray projection image depicting the spine from the anteroposterior direction. A surgical object (1.SO) is positioned such that its tip is on the bone surface of vertebra T8 (1.T8). Two different 3D positions of the surgical object (1.P1 and 1.P2 respectively) and two outlines of the surgical object in 3D orientations with substantially the same outline (marked as white solid lines 1.O1 and 1.O2 respectively) are shown. The tip of the surgical object in 3D space is on the epipolar line defined by the drill tip and the focal point of the X-ray device, i.e., the epipolar line of the tip of the surgical object. Depending on the 3D position of the tip of the surgical object, the 3D orientation is adjusted so that the projected outline of the surgical object fits its appearance in the 2D X-ray image. It is emphasized that there are an infinite number of combinations of 3D positions and corresponding 3D orientations that lead to substantially the same outline of the surgical object, two of which are depicted in FIG. 1.

[0050] Figure 2 shows a slice of the same 3D scan of the spine as in Figure 1, depicting a sagittal section. The white line (2.EL) indicates all possible 3D positions of the tip of the surgical object corresponding to the epipolar line at the tip of the surgical object in Figure 1. Two points on the line are selected, one point (2.P1) where the tip of the surgical object is on the bone surface of vertebra T8 (2.T8), and the other point (2.P2) at a specific distance from the bone surface. Depending on the 3D position of the surgical object, it can be clearly visually recognized that the 3D postures of the surgical object are different, but both result in almost the same outline in the X-ray projection image of Figure 1.

[0051] Figure 3 shows a slice of the same 3D scan as in Figure 2, depicting a body-axis section. The white line (3.EL) indicates all possible 3D positions of the tip of the surgical object corresponding to line 2.EL in Figure 2. Similar to Figure 2, two points on the line are selected, one point (3.P1) where the tip of the surgical object is on the bone surface of vertebra T8 (3.T8), and the other point (3.P2) at a specific distance from the bone surface. Here too, it can be visually recognized that the 3D postures of the surgical object are different, but both result in almost the same outline in the X-ray projection image of Figure 1.

[0052] In other words, Figures 1 - 3 show two different 3D constellations in two different drill poses, resulting in the same (or almost the same) X-ray projection image. The two drill constellations differ in imaging depth (distance from the image plane, i.e., from the X-ray receiver) and tilt (i.e., 3D posture). However, if the ambiguity regarding the imaging depth can be eliminated (by some prior information), the drill pose with respect to the patient's anatomical structure can be uniquely determined.

[0053] The present disclosure teaches eliminating ambiguity regarding imaging depth by establishing a correspondence between an anchor point and a target point (the position in a 3D coordinate system defined by a 3D data set is known) of a surgical object using prior information that the target point and the anchor point are on or at a defined distance from the bone surface. This can be done based on a single X-ray image.

[0054] Alternatively, it is also possible to eliminate ambiguity regarding imaging depth without using a target point. According to one embodiment, the tip of the drill (i.e., the anchor point) is placed on the bone surface near the intended drilling path (i.e., within the region of interest), and an X-ray image is acquired. The position of the drill tip is detected in the X-ray image. For the imaging direction to the region of interest determined based on the DRR matching of the 3D data set and the X-ray image, a virtual projection onto the X-ray image is made at the possible drill tip positions in 3D space located on the detected anchor point, and a line (so-called epipolar line) in the 3D coordinate system defined by the 3D data set is determined. Since it is known in advance that the drill tip is on the bone surface, by determining a point on the line that intersects the bone surface (i.e., is on the bone surface), the position of the drill tip in the coordinate system defined by the 3D data set can be found. Detection of the bone surface along the line can be automatically performed, for example, by evaluating the Hounsfield values in the 3D data set along (or near) the line and searching for strong gradients. Of course, if local segmentation of the bone surface within the relevant region is already available, this can be used for the intersection with the line.

[0055] The accuracy of determining the spatial relationship between the surgical object and the region of interest can be enhanced by acquiring additional X-ray images from the current imaging direction and / or different imaging directions.

[0056] After a surgical object has been inserted into a patient, i.e., after the anchor point is no longer located on the bone surface, it is also possible to determine the relative 3D spatial relationship between the surgical object and the region of interest. This is interesting, for example, when drilling. In such a case, at a first time point, an X-ray image is acquired, and from this, the starting point of the drilling in the coordinate system of the 3D data is determined, which is considered to be the 3D position of the drill tip (i.e., the anchor point) at the first time point. After advancing the tool into the patient's body, at a second time point, a further X-ray image is acquired. Here, based on the assumption that the drill axis determined at the second time point passes through the starting point of the drilling determined at the first time point, the ambiguity regarding the 3D drill pose can be resolved. In this assumption, drilling must actually start at the anchor point determined at the first time point. This assumption can be particularly easily justified when using a surgical robot, as unintended movements of the surgical object are excluded and patient movements are detected by single image registration.

[0057] This procedure can be repeated, and X-ray images from other imaging directions can also be acquired to increase accuracy.

[0058] According to one embodiment, such a system may be particularly useful for spinal surgery. The following three workflows are examples of how this system can be used to perform pedicle screw drilling in spinal fixation. The first workflow (see Figure 4) is an extended workflow for increased accuracy and requires multiple X-ray images from different imaging directions. The second workflow (see Figure 5) is a shorter workflow without changing the imaging direction, and the third workflow (see Figure 6) is a workflow without planning a target point.

[0059] Exemplary workflow for performing pedicle screw drilling (extended workflow) (see Figure 4)

[0060] Note that the step numbers in the workflow of FIG. 4 are a combination of "1.", which indicates that the workflow is the first of three workflows, and the actual step numbers used in the following description of the workflow.

[0061] 1. The surgeon and / or the system performs preoperative planning on the 3D data set by determining the target path, i.e., the planned drilling trajectory, within the 3D data set (e.g., a CT scan or a 3D X-ray scan). Here, the target point is the starting point of the intended drilling trajectory located on the bone surface of the pedicle, and the target end point is the end point of the intended drilling trajectory.

[0062] 2. The surgeon places the drill tip on the surface of the spine near the pedicle to be drilled (if the relative position between the drill and the bone is already known, the drill tip can be placed more accurately).

[0063] 3. The surgeon acquires an X-ray image (e.g., in the imaging direction approximately anteroposterior (AP)), and preferably holds the drilling machine in such a manner that the surgeon's hand does not enter the X-ray beam. If the imaging direction is such that the surgeon's hand enters the X-ray beam, the system may provide an instruction for a different imaging direction.

[0064] 4. The system detects the position of the drill tip (i.e., the anchor point) in the X-ray image. The system determines the imaging direction to the region of interest in the X-ray image by calculating DRRs (digitally reconstructed radiographs) in multiple imaging directions from the 3D dataset. The DRR that best matches the X-ray image is employed to determine the imaging direction. It is not necessary to constrain the DRR to the region of interest. The system can use a weighting function to more accurately match important regions within the 2D region of interest (e.g., when the image quality is poor), and the 2D region of interest is determined by the position of the drill tip in the X-ray image. The weighting function can also depend on the type of C-arm (flat panel / image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D dataset (pre-operative / intra-operative). The system determines a virtual projection of the target point onto the X-ray image based on the imaging direction, and the projection can be displayed as an overlay on the X-ray image.

[0065] 5. If the 2D distance between the drill tip (i.e., the anchor point) in the X-ray image and the virtual projection of the target point is below a threshold (e.g., 2 mm, and the threshold can depend on the type of vertebra), the system proceeds to step 8. Otherwise, the system gives an instruction to relocate the drill tip to approach the virtual projection of the target point. The surgeon a. follows the instruction and returns to step 3, or b. continues with the current drill tip position and proceeds to step 6, and decides.

[0066] 6. While confirming that the drill tip position has not moved compared to the previous X-ray image, the surgeon acquires an X-ray image from a different imaging direction (e.g., lateral, or a direction between AP and lateral, or a direction between AP and the body axis direction).

[0067] 7. The system determines a new imaging direction by performing a new DRR matching between the 3D data set and the current X-ray image as described above. Thereby, the current X-ray image is registered with the previous X-ray image. The system detects the drill tip in the current X-ray image and, based on epipolar geometry, uses the known 3D model of the drill to determine the 3D position of the drill tip in the coordinate system of the 3D data set. If it can be guaranteed that the drill has not moved or tilted during the acquisition of the X-ray images in steps 3 and 6 (for example, when a robot holds the drill), the matched drill can be used to improve the accuracy of image registration, and in this case, proceed to step 9.

[0068] 8. The system determines the 3D drill tip position in the coordinate system of the 3D data set based on the detected drill tip in the X-ray image by applying the knowledge that both the drill tip and the target point are on the bone surface. If the drill tip has a remaining 2D distance to the target point, a local model of the vertebral surface in the vicinity of the target point can be considered in the determination of the 3D drill tip position. If the vertebra to be drilled (i.e., which vertebral level, for example, L3, L4, etc.) has already been classified, the local model can be derived from the model of the vertebra.

[0069] 9. Based on the determined 3D drill tip position in the coordinate system of the 3D data set, the system determines the 3D position and orientation of the drill in the coordinate system of the 3D data set.

[0070] 10. If the deviation between the 3D drill orientation and the target path is below a threshold (for example, less than 2°), proceed to step 12. Otherwise, the system gives an instruction to adjust the drill angle.

[0071] 11. The surgeon adjusts the drill angle and returns to step 3 (no need to change the imaging direction) or proceeds directly to step 13 to continue / start drilling.

[0072] 12. The system calculates the (remaining) drilling depth and gives the corresponding drilling instruction.

[0073] 13. The surgeon follows the instruction (e.g., drills by a given distance). Whenever confirmation of the drilling trajectory is desired or advised by the system (e.g., after drilling a specific distance or when drilling near an important anatomical structure), a new X-ray image can be acquired (without the need to change the imaging direction).

[0074] 14. The system performs matching, detection, etc. as described above. Based on the actual starting point of drilling (which can be, for example, the position of the drill tip shown in the X-ray image when the first drilling instruction is given), the system determines the 3D position and 3D orientation of the drill.

[0075] 15. If the remaining drilling distance is less than a threshold (e.g., 1 mm), proceed to step 17. If the deviation between the 3D drill orientation and the target path is less than a threshold (e.g., 1°), return to step 12. Otherwise, the system gives an instruction to adjust the drill angle. If the drilling is too deep, the system may provide a corresponding warning.

[0076] 16. The surgeon follows the instruction, acquires a new X-ray image, and returns to step 14.

[0077] 17. The current pedicle drilling procedure is completed.

[0078] 18. To perform the next pedicle drilling, return to step 2.

[0079] It should be noted again that all references to "surgeon" in this workflow may refer not only to human users but also to robotic surgeons, mechanical support devices, or similar apparatuses.

[0080] Generally, and in particular, in addition to each of the exemplary workflows disclosed herein, the system can also provide support for making an incision. This can be particularly useful when a surgical robot or robotic arm is used. According to one embodiment, a surgical instrument (e.g., a surgical scalpel) is aligned with a target trajectory, but the distance from the target point is large enough that the surgical instrument has not yet touched the skin. Using a soft tissue protection sleeve, the robot or robotic arm can move the surgical instrument along the target trajectory towards the target point until it encounters some resistance (i.e., touches the skin). The resistance can be automatically detected, for example, by a pressure sensor. The incision may be made through the soft tissue protection sleeve. After making the incision, the surgical instrument can be exchanged (e.g., switched to a drill while keeping the soft tissue protection sleeve in place) and advanced until the surgical instrument encounters greater resistance (i.e., touches the bone surface). The resistance can be automatically detected, for example, by a pressure sensor. After completion of drilling one pedicle, the procedure can be repeated for the next pedicle.

[0081] It may be useful if the robot or robotic arm can automatically switch the surgical instrument (possibly within a soft tissue protection sleeve), advance or retract the soft tissue protection sleeve, and drill through the soft tissue protection sleeve (possibly using a vibratory drilling mode). It may be advantageous to specially design a soft tissue protection sleeve or adapter suitable for the robot. In the design of such a sleeve, detectability in X-ray images is also considered. For example, it may be advantageous to make the sleeve of a radiation-transparent material or (e.g., in the region covering the tip of the drill) partially of a radiation-transparent material so that surgical objects (e.g., the drill and especially its tip) can be seen in X-ray images even when covered by the sleeve. It may also be advantageous to incorporate specific features into the surgical objects and / or the sleeve used to make it easier to detect the surgical objects in X-ray images.

[0082] According to one embodiment, by using a surgical robot, it may be possible to cope with the movement between individual vertebrae caused by the patient's breathing. This breathing can be modeled and detected by a pressure sensor. The robot can keep the pressure on the vertebra constant when not drilling.

[0083] This procedure may also be applied to determine the position of the chisel, for example, when the midpoint of the chisel is used as the anchor point during osteotomy.

[0084] A similar procedure is also applicable when using a soft tissue protection sleeve made of a non-radiopaque material and the anchor point (e.g., the tip of the drill) cannot be seen in the X-ray image. In such a case, instead of using the (invisible) anchor point, a virtual anchor point can be used. This virtual anchor point can be determined based on the assumption that there has been no movement of the anchor point between the generation of the current X-ray image and the generation of the previously acquired X-ray image, based on the previously acquired X-ray image when the anchor point was visible. Based on the current X-ray image, a plane is defined by the axis of the sleeve and the center of the X-ray beam. The virtual anchor point can be determined by orthogonally projecting the anchor point of the previous X-ray image onto that plane. The virtual anchor point may also be determined by selecting the point closest to a contour (e.g., the intersection of that plane with a model of the bone or a partial 3D segmentation of the bone surface) within that plane from the anchor point of the previous X-ray image so that the virtual anchor point is surely placed on the bone surface.

[0085] Another use of the procedure can be tumor resection. In this case, an MRI scan obtained preoperatively (e.g., including the outline of the tumor or the planned resection volume) is fused with a 3D X-ray scan obtained preoperatively or intraoperatively, and one or more target points are identified in the MRI scan and / or the 3D X-ray scan. The surgical instrument can be a resection instrument having anchor points. By determining the relative 3D position and 3D orientation of the resection instrument with respect to the outline of the tumor (or the planned resection volume), accurate 3D navigation instructions can be provided even if several target points are defined during planning. The resection may be performed at a specific distance from the target point. The use of a robot may also improve accuracy. The robot can receive relative 3D position information from, for example, an internal sensor device, and thus the 3D position and 3D orientation of the instrument with respect to the resection volume are available at any time after the final decision based on the acquisition of X-ray images. Verification of this information can be done at any time by acquiring additional X-ray images without using anchor points or target points. Based on the information available from the position sensor device and the final decision on the 3D position and 3D orientation of the instrument with respect to the resection volume based on the last X-ray image, a virtual X-ray image is generated assuming that the imaging direction has not been changed. After acquiring additional X-ray images from the same imaging direction, the algorithm determines whether the additional X-ray images are sufficiently similar (the position information is still valid) or not (there may have been unknown movement). In the latter case, new images may be required along with the determination of anchor points and target points.

[0086] This procedure can also be combined with a real-time navigation and tracking system. Since this procedure is redundant with existing technologies and provides a high degree of diversification, such a combination makes the navigation very robust, and this level of safety may enable autonomous robotic surgery.

[0087] Exemplary workflow for performing pedicle screw drilling (without changing the imaging direction) (see Figure 5)

[0088] Note that the step numbers in the workflow of Figure 5 are a combination of "2.", indicating that the workflow is the second of three examples, and the actual numbers of the steps used in the following description of the workflow.

[0089] 1. The surgeon and / or the system perform preoperative planning on the 3D data set by determining target points within the 3D data set (e.g., a CT scan or a 3D X-ray scan). Here, the target points are the starting points of the intended drilling trajectories located on the bone surface of the pedicles.

[0090] 2. The surgeon places the drill tip on the surface of the spine near the pedicle to be drilled (if the relative position between the drill and the bone is already known, the drill tip can be placed more precisely).

[0091] 3. The surgeon acquires an X-ray image (e.g., in an imaging direction generally in the anteroposterior (AP) direction), and preferably holds the drilling machine in such a manner that the surgeon's hand does not enter the X-ray beam. If the imaging direction is such that the surgeon's hand enters the X-ray beam, the system may provide an indication of a different imaging direction.

[0092] 4. The system detects the position of the drill tip (i.e., the anchor point) in the X-ray image. The system determines the imaging direction to the region of interest in the X-ray image (which may already have been done before acquisition of the X-ray image) by calculating DRRs (digitally reconstructed radiographs) in a plurality of imaging directions from the 3D data set. The DRR that best matches the X-ray image is employed to determine the imaging direction. It is not necessary to constrain the DRR to the region of interest. The system can use a weighting function to more accurately match important regions within the 2D region of interest (e.g., when the image quality is poor), and the 2D region of interest is determined by the position of the drill tip in the X-ray image. The weighting function can also depend on the type of C-arm (flat panel / image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D data set (pre-operative / intra-operative). The system determines a virtual projection of the target point onto the X-ray image based on the imaging direction, and the projection can be displayed as an overlay on the X-ray image.

[0093] 5. If the 2D distance between the drill tip (i.e., the anchor point) in the X-ray image and the virtual projection of the target point is below a threshold (e.g., 1 mm, the threshold may depend on the type of vertebra), the system proceeds to step 6. Otherwise, the system gives an instruction to relocate the drill tip to approach the virtual projection of the target point. The surgeon follows the instruction and returns to step 3.

[0094] 6. The system determines the 3D drill tip position in the coordinate system of the 3D data set based on the detected drill tip in the X-ray image by applying the knowledge that both the drill tip and the target point are on the bone surface. If the drill tip has a remaining 2D distance to the target point, a local model of the vertebra surface in the vicinity of the target point can be taken into account in determining the 3D drill tip position. If the vertebra to be drilled (i.e., which vertebra level, e.g., L3, L4, etc.) has already been classified, the local model can be derived from the model of the vertebra.

[0095] 7. Based on the determined 3D drill tip position in the coordinate system of the 3D dataset, the system determines the 3D position and orientation of the drill in the coordinate system of the 3D dataset. The system can display the current position in different views based on the 3D dataset, such as axial and sagittal images, and extend the trajectory of the drill to better visualize the corresponding drilling path.

[0096] 8. The surgeon adjusts the drill angle and returns to step 3 (no need to change the imaging direction) or proceeds directly to step 9 to start / continue drilling.

[0097] 9. Whenever it is desired or advised by the system to confirm the drilling trajectory (e.g., after drilling a certain distance or when drilling near important anatomical structures), a new X-ray image can be acquired (no need to change the imaging direction).

[0098] 10. The system performs matching, detection, etc. as described above. Based on the actual starting point of drilling (which can be, for example, the position of the drill tip shown in the X-ray image when the first drilling instruction is given), the system determines the 3D position and 3D orientation of the drill. The system displays the current position of the drill in different views based on the 3D dataset, such as axial and sagittal images. If the surgeon does not consider the current position appropriate, the angle is corrected (e.g., while the drill bit is operating) and the process proceeds to step 9.

[0099] 11. The surgeon determines whether to complete the current pedicle drilling procedure.

[0100] 12. To perform the next pedicle drilling, return to step 2.

[0101] Exemplary workflow for performing pedicle screw drilling (without planning of target points) (see Figure 6)

[0102] Similar to the previous workflow, the step numbers in the workflow of FIG. 6 are a combination of "3.", indicating that the exemplary workflow is the third of three, and the actual numbers of the steps used in the following description of the workflow.

[0103] 1. The surgeon places the drill tip on the surface of the spine near the pedicle to be drilled.

[0104] 2. The surgeon acquires an X-ray image (e.g., in an imaging direction generally in the anteroposterior (AP) direction), and preferably holds the drilling machine in such a manner that the surgeon's hand does not enter the X-ray beam. If the imaging direction is such that the surgeon's hand enters the X-ray beam, the system may provide an indication of a different imaging direction.

[0105] 3. The system detects the position of the drill tip (i.e., the anchor point) in the X-ray image. The system determines the imaging direction to the region of interest in the X-ray image (which may already have been done before acquisition of the X-ray image) by calculating DRRs (digitally reconstructed radiographs) in a plurality of imaging directions from the 3D dataset. The DRR that best matches the X-ray image is employed to determine the imaging direction. It is not necessary to constrain the DRR to the region of interest. The system can use a weighting function to more accurately match important regions within the 2D region of interest (e.g., when the image quality is poor), and the 2D region of interest is determined by the position of the drill tip in the X-ray image. The weighting function may also depend on the type of C-arm (flat panel / image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D dataset (pre-operative / intra-operative). The system determines a line within the 3D dataset defined by the anchor point and the center of the beam. The system determines a point on this line that intersects the bone surface (e.g., starting at the most posterior point of the line, observing the voxels on this line, and determining the voxel with the highest gray-scale gradient on this line and possibly around this point). This point defines the position of the anchor point in the 3D dataset.

[0106] 4. Based on the determined 3D drill tip position in the coordinate system of the 3D dataset, the system determines the 3D position and orientation of the drill in the coordinate system of the 3D dataset. The system can display the current position in different views, e.g., axial and sagittal images, based on the 3D dataset, and can extend the trajectory of the drill to better visualize the corresponding drilling path.

[0107] 5. The surgeon adjusts the drill tip position and / or angle and returns to step 2 (without the need to change the imaging direction) or proceeds directly to step 6 to start / continue drilling.

[0108] 6. Whenever it is desired to check the drilling trajectory or it is advised by the system (e.g., when using a robot for drilling), for example, after drilling a certain distance or when drilling near an important anatomical structure, a new X-ray image can be acquired (without the need to change the imaging direction).

[0109] 7. The system performs matching, detection, etc. as described above. Based on the actual starting point of drilling (which can be, for example, the position of the drill tip shown in the X-ray image when the first drilling instruction is given), the system determines the 3D position and 3D orientation of the drill. The system displays the current position of the drill in different views, such as axial and sagittal images, based on the 3D dataset. If the surgeon does not recognize that the current position is appropriate, (e.g., while the drill bit is operating), the angle is corrected and the process proceeds to step 6.

[0110] 8. The surgeon determines whether to complete the current pedicle drilling procedure.

[0111] 9. To perform the next pedicle drilling, return to step 2.

[0112] Examples of methods are provided below, and these methods are implemented as computer program products.

[0113] 1. An example of a computer program product that, when executed by a processing unit of a system, causes the system to determine the 3D position and 3D orientation of object 1 with respect to object 2 at a first point in time, determine the 3D position and 3D orientation of object 1 with respect to object 2 at a second point in time based on information received from a real-time tracking system, receive a 2D X-ray image generated by an X-ray imaging device at the second point in time depicting at least a part of object 1 and at least a part of object 2, Receive either 3D data including object 1 or a 3D model of object 1, Receive either 3D data including object 2 or a 3D model of object 2, (i) 3D data including object 1 or a 3D model of object 1, (ii) 3D data including object 2 or a 3D model of object 2, and (iii) The 3D position and 3D orientation of object 1 relative to object 2 at a second point in time based on information received from a real-time tracking system, Generate a virtual projection image based on the above, Compare the 2D X-ray image with the virtual projection image, An example of a computer program product including instructions.

[0114] 2. The 3D position and 3D orientation of object 1 relative to object 2 at a second point in time based on information received from a real-time tracking system are verified based on the comparison between the 2D X-ray image and the virtual projection image, the computer program product according to Example 1.

[0115] 3. The comparison is based on a similarity threshold based on different weightings of different 2D regions of the images, the computer program product according to Example 2.

[0116] 4. The imaging direction of the 2D X-ray image is Generation of a plurality of virtual projection images of object 1 and selection of a virtual projection image from a plurality of images that best fit object 1 in the 2D X-ray image, The assumption that the imaging direction of the X-ray imaging device is the same as that determined at the previous point in time, Tracking the movement of the X-ray imaging device by a real-time tracking system, Determined based on at least one of the group consisting of, the computer program product according to any one of Examples 1 to 3.

[0117] 5. The determination of the 3D position and 3D orientation of object 1 with respect to object 2 at the first point in time is based on at least one of the group consisting of obtaining a plurality of surface points, obtaining a plurality of 2D X-ray images with a tracking X-ray imaging device, obtaining an intraoperative 3D scan with a tracked imaging device, obtaining at least one 2D X-ray image including an object having anchor points, the computer program product according to any one of Examples 1 to 4.

[0118] 6. The real-time tracking system is at least one of the group consisting of a sensor device for the position and / or movement of a robot, a navigation system equipped with an optical tracker, a navigation system equipped with an infrared tracker, a navigation system equipped with EM tracking, a navigation system using a 2D camera, a navigation system using LIDAR, a navigation system using a 3D camera, a navigation system including wearable tracking elements such as augmented reality glasses, the computer program product according to any one of Examples 1 to 5.

[0119] 7. Based on another determination of the 3D position and 3D orientation of object 1 with respect to object 2 at the second point in time, the 3D position and 3D orientation of object 1 with respect to object 2 at the second point in time are further verified, and the other determination is based on a single 2D X-ray image considering a specific point in one of the objects, the computer program product according to Example 2.

[0120] A system including a processing unit configured to execute the computer program product according to any one of the foregoing examples further includes a real-time navigation and tracking system, and the real-time navigation and tracking system may be configured to provide the 3D position and 3D orientation of the first object with respect to the second object. The system may further include a robot device for holding and moving one of the objects. The system may further include an X-ray imaging device.

[0121] A method for verifying the 3D position and 3D orientation of object 1 with respect to object 2 is as follows: Determining the 3D position and 3D orientation of object 1 with respect to object 2 at a first point in time; Determining the 3D position and 3D orientation of object 1 with respect to object 2 at a second point in time based on information received from a real-time tracking system; Receiving a 2D X-ray image generated by an X-ray imaging device at the second point in time, depicting at least a part of object 1 and at least a part of object 2; Receiving either 3D data containing object 1 or a 3D model of object 1; Receiving either 3D data containing object 2 or a 3D model of object 2; (i) 3D data containing object 1 or a 3D model of object 1, (ii) 3D data containing object 2 or a 3D model of object 2, and (iii) the 3D position and 3D orientation of object 1 with respect to object 2 at the second point in time, Generating a virtual projection image based on the above; Comparing the 2D X-ray image with the virtual projection image; may be included.

[0122] Although several embodiments have been shown and described in detail in the drawings and the above description, such illustrations and descriptions should be considered illustrative or exemplary and not restrictive, and the present invention is not limited to the disclosed embodiments.

[0123] Upon consideration of the drawings, the present disclosure, and the appended claims, those skilled in the art will understand and be able to implement other variations of the disclosed embodiments in practicing the claimed invention. In the claims, the term "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plural number. A single processor or other unit may perform the functions of several items recited in the claims.

[0124] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be advantageously used.

Claims

**Claim 1** A computer program product that, when executed by a processing unit of a system, causes the system to receive an X-ray image that is a 2D projection image depicting at least a portion of a surgical object including an anchor point and a region of interest within a patient's anatomical structure, generated at a first time point, receive a 3D dataset that describes the region of interest and defines a 3D coordinate system, receive a 3D model of the surgical object, wherein the position of the anchor point is known, determine the 2D position of the anchor point in the X-ray image, determine an imaging direction to the region of interest based on the X-ray image, determine the 3D position of the anchor point in the 3D coordinate system at the first time point i. based on the 2D position of the anchor point in the X-ray image, and ii. the imaging direction to the region of interest, and based on this, determine the 3D position and 3D orientation of the surgical object with respect to the region of interest at the first time point i. based on the 3D model of the surgical object, and ii. the 3D position of the anchor point in the 3D coordinate system at the first time point, and based on this, determine the 3D position and 3D orientation of the surgical object with respect to the region of interest at the first time point, and a computer program product including instructions. **Claim 2** The system is further caused to determine a line in the 3D coordinate system, the virtual projection onto the X-ray image of which includes a point located at the anchor point in the X-ray image, determine points of the line located on the bone surface within the region of interest, and the determination of the 3D position of the anchor point in the 3D coordinate system at the first time point is further based on the points of the line located on the bone surface. The computer program product according to claim 1. **Claim 3** The 3D dataset includes target points whose positions in the 3D coordinate system are known. The computer program product further includes instructions that cause the system to determine the virtual projection of the target points onto the X-ray image. The determination of the 3D position of the anchor point in the 3D coordinate system at the first time point is further based on the 2D position of the virtual projection of the target points with respect to the 2D position of the anchor point in the X-ray image at the first time point. The computer program product according to any one of claims 1 or 2. **Claim 4** ​ The computer program product according to claim 3, wherein the 2D position of the anchor point in the X-ray image at the first time point and the 2D position of the virtual projection of the target point in the X-ray image are treated as the same.

5. The determination of the 3D position and 3D orientation of the surgical object with respect to the region of interest is further based on the distance of the target point from the bone surface within the region of interest and the distance of the anchor point from the bone surface within the region of interest, the computer program product according to any one of claims 3 or 4.

6. The computer program product according to claim 5, wherein the distance of the target point from the bone surface within the region of interest is treated as zero, and the distance of the anchor point from the bone surface within the region of interest is treated as zero.

7. The system, receives a new X-ray image that is a 2D projection image depicting at least a part of the surgical object and the region of interest, generated at a second time point, determines the imaging direction of the new X-ray image, further includes an instruction, and the determination of the 3D position and 3D orientation of the surgical object with respect to the region of interest at the second time point is i. the 3D model of the surgical object, ii. the imaging direction of the new X-ray image, iii. the 3D position of the anchor point in the 3D coordinate system at the first time point, and iv. the assumption that the axis of the surgical object at the second time point passes through the 3D position of the anchor point in the 3D coordinate system at the first time point, based on the computer program product according to any one of claims 1 to 6.

8. The system, receives a new X-ray image that is a 2D projection image depicting at least a part of the surgical object and at least a part of the region of interest, generated at a second time point, determines the imaging direction of the new X-ray image, and determines the expected 3D position and 3D orientation of the surgical object with respect to the region of interest at the second time point (i) the 3D position and 3D orientation of the surgical object with respect to the region of interest at the first time point, and (ii) information regarding the movement of the surgical object between the first time point and the second time point, based on this. Based on the determined imaging direction of the new X-ray image, generate a virtual projection image including the surgical object at the predicted 3D position and 3D pose of the surgical object with respect to the region of interest at the second time point. Compare the virtual projection image with the new X-ray image to verify the 3D position and 3D pose of the surgical object with respect to the region of interest at the second time point. The computer program product according to any one of claims 1 to 6, further including instructions.

9. In the system, Receive another X-ray image that is a 2D projection image depicting at least a part of the surgical object and the region of interest, generated at a third time point from another imaging direction different from the imaging direction of the new X-ray image generated at the second time point. Determine the other imaging direction of the other X-ray image. Determine the 2D position of the anchor point in the other X-ray image. Further including instructions, and the determination of the 3D position and 3D pose of the surgical object with respect to the region of interest at the third time point is i. The imaging direction of the new X-ray image at the second time point, ii. The other imaging direction of the other X-ray image at the third time point, iii. The 2D position of the anchor point in the other X-ray image, and iv. The assumption that the 3D position of the anchor point at the second time point is the same at the third time point. Based on this, the computer program product according to any one of claims 7 and 8.

10. The computer program product according to any one of claims 1 to 9, wherein the distortion of the X-ray image is corrected based on at least one of the 3D dataset, the 3D model of the surgical object, and another surgical object with a known geometric shape.

11. The computer program product according to any one of claims 1 to 10, further including instructions to cause the system to provide the user with an indication of how to adjust at least one of the position of the surgical object and the 3D pose of the surgical object.

12. The computer program product according to any one of claims 1 to 11, wherein knowledge of the position of the target point is automatically obtained.

13. Determining the relative 3D position and 3D orientation of the surgical object with respect to the region of interest is further based on a local model of the bone surface in the vicinity of the target point, the computer program product according to any one of claims 3 to 12.

14. A system comprising a processing unit configured to execute a computer program product according to any one of claims 1 to 13.

15. A robotic device to which the surgical object is attached, A real-time navigation and tracking system, Further comprising, wherein the 3D position and 3D orientation of the surgical object with respect to the region of interest are determined based on both a computer program product according to any one of claims 1 to 13 and information received from the real-time navigation and tracking system, the system according to claim 14.

Images