System and method for a two-step registration procedure enabling complete robotic surgical intervention

Abstract

The present invention provides a system and method for a two-step registration procedure that enables complete surgical intervention by a robot. [Solution] A system for intraoperative navigation is provided, comprising a processing unit and a tracking device for real-time or near-real-time tracking of an object in a 3D coordinate system, wherein the processing unit is configured to receive an X-ray image depicting an anatomical structure including at least one bone, and to determine an incomplete registration of the position of the anatomical structure in the 3D coordinate system. The incomplete registration is determined based on a 3D dataset of the received anatomical structure and a plurality of virtual X-ray images generated from different directions based on the 3D dataset of the anatomical structure. The incomplete registration can be understood as an approximation of the position of the anatomical structure. Based on the incomplete registration, instructions for the navigation of the device are provided.

Description

Technical Field

[0001] The present invention relates to the field of computers and robot-assisted surgery. Further, the present invention relates to systems and methods for providing information regarding surgical objects and anatomical structures based on CT data, or another type of 3D X-ray scan, and X-ray images. Further, the present invention relates to a system for treating a patient by a surgical robot. The method may be implemented as a computer program executable on a processing unit of the system.

Background Art

[0002] Spinal surgery, particularly spinal fixation, is a commonly performed surgical procedure. However, spinal surgery also poses a significant risk to the patient. Incorrect drilling on the spine can damage the spinal cord, which can cause serious damage to the nerves or the meninges of the spinal cord, leading to chronic pain, permanent paralysis, incontinence, or sexual dysfunction. To reduce these risks associated with incorrect drilling, computer-assisted navigation is already used to some extent in spinal surgery. Robot assistance supports the surgeon in accurately moving surgical instruments, supports that drilling is performed at the correct position, and that pedicle screws are properly placed, etc. This involves determining the exact relative 3D position between a surgical instrument (such as a drill) and the patient (e.g., with respect to a desired drill trajectory).

[0003] Almost all existing musculoskeletal robot systems require additional procedural steps and devices such as 3D cameras, trackers, reference bodies, etc. In navigated spinal surgery, most current systems use optical tracking, where a dynamic reference frame is attached to the spine (to track any movement of the patient) and a reference body is attached to the surgical instrument.

[0004] Existing musculoskeletal robotic systems are prone to errors. The long chain of errors in the navigation workflow—registration, instrument calibration, and real-time tracking—means that the accuracy of the navigated robotic system can only be as high as the weakest point in the chain. Therefore, surgeons must always ensure the accuracy of navigation instructions and robotic support positions through uncertain verification procedures such as tactile and landmark checks.

[0005] Robot-assisted surgical systems are gaining popularity due to the perceived precision they offer. However, since robots essentially operate based on instructions provided by navigation systems, their precision is only as close to the information provided by those navigation systems as possible. Existing navigation systems are prone to errors, so current robotic surgeons are not trusted to autonomously perform any surgical procedure. Rather, robotic surgeons are simply automated holding arms that hold instruments. The actual drilling still needs to be performed by the surgeon.

[0006] It is desirable to have a navigation system that can reliably determine the actual 3D position and orientation of surgical objects (e.g., instruments or implants) relative to the patient. This must be reliably possible even when the patient or the patient's anatomical structures are moving, such as during surgery on a moving thoracic spine, which may occur due to the patient's breathing or other patient movements, such as during drilling procedures like distally fixing a long nail into a long bone, or during screw insertion procedures like pedicle screws or pelvic screws, or due to movement caused by applied pressure.

[0007] It is even more desirable to have the option to repeat registration as quickly as possible in case of any changes that may negatively affect existing registrations. Also, lead protective suits are heavy and therefore a burden on surgeons and operating room (OR) staff, so it may be desirable to eliminate the need to wear these suits in OR settings. Therefore, reducing the number of required registrations to a single-digit number per surgery would be beneficial for both OR staff and patients.

[0008] While robotic systems may be particularly useful in spinal surgery, they can also be used in many other surgical procedures where the spatial relationship of surgical objects or implants to the patient must be precisely determined. When used in combination with the conventional navigation systems described above, redundant navigation information can be provided, thus making navigation extremely robust, and this level of safety may enable autonomous robotic surgery. [Overview of the Initiative]

[0009] At least one of the above-mentioned problems is mitigated or resolved by the subject matter of the independent claim. Further embodiments are described in the dependent claims.

[0010] Systems and methods are proposed that do not require a reference body or tracker to determine the 3D spatial relationship or 3D position of a surgical object to a part of a patient's anatomical structure. However, it will be understood that some of these methods may be applied to improve systems that use a reference body or tracker. The system may comprise a processing unit and a computer program product containing instructions that can be executed on the processing unit. The computer program product may carry out a computer implementation method.

[0011] Generally, an intraoperative navigation system may include a processing unit and tracking device for real-time or near-real-time tracking of objects in a 3D coordinate system, the processing unit being configured to receive X-ray images depicting an anatomical structure including at least one bone, and to determine an incomplete registration of the position of the anatomical structure in the 3D coordinate system.

[0012] Incomplete registration is determined based on the received 3D dataset of the anatomical structure and multiple virtual X-ray images generated from different directions based on that 3D dataset. Incomplete registration can be understood as an approximation of the location of the anatomical structure.

[0013] According to the method, the position of an anatomical structure in a 3D coordinate system can be determined based on a single X-ray image. The method includes the steps of receiving an X-ray image depicting an anatomical structure, receiving a first plurality of virtual X-ray images, wherein the virtual X-ray images among the first plurality of virtual X-ray images are generated as projection images of a 3D dataset of anatomical structures having a wide distribution of different imaging directions, identifying the image features of the received X-ray image, identifying the image features of the said virtual X-ray image among the first plurality of virtual X-ray images, and performing 2D-2D matching between the image features of the virtual X-ray image among the first plurality of virtual X-ray images and the image features of the received X-ray image by applying at least one of the group consisting of image rotation, image zoom, and translation in the image plane to determine one of the first plurality of virtual X-ray images that provides image features that best match the image features of the received X-ray image, wherein the determined virtual X-ray image among the first plurality of virtual X-ray images is used to determine an approximation of the position of the anatomical structure.

[0014] Based on such approximations or incomplete registrations, the object may be navigated for (i) soft tissue incision, (ii) providing a soft tissue gateway to at least one bone, (iii) positioning the object on the outside or surface of at least one bone, and / or (iv) positioning further objects on the outside or surface of at least one bone, the further surgical objects being registered and tracked in a 3D coordinate system by a tracking device.

[0015] After object navigation, the complete registration of the position of the anatomical structure in the 3D coordinate system may be determined based on at least one of the following: (i) additional X-ray images generated in an imaging direction different from the imaging direction of previously received X-ray images; (ii) utilization of the endoscope, where the position of the endoscope is registered in the 3D coordinate system, the endoscope images are images of anatomical structures, the image content is processed and compared with a 3D dataset; (iii) utilization of the object's points, where the object's position is changed so that a point touches the surface of at least one bone at a given point; (iv) utilization of the object's points, where the object's position is changed so that a point touches the surface of at least one bone at an unknown point, and the application of an undefined trajectory with undefined degrees of freedom to determine the intersection of the surface and the trajectory at the unknown point; (v) utilization of further object's points, where further object's positions are changed so that further object's points touch the surface of at least one bone at a given point; (vi) utilization of further object's points, where further object's positions are changed so that further object's points touch the surface of at least one bone at an unknown point, and the application of an undefined trajectory with undefined degrees of freedom to determine its intersection with the surface of the anatomical structure.

[0016] The spatial relationships of objects can be understood as constraints on parameterization having at least one degree of freedom to partially or completely describe the position and orientation between two 3D coordinate systems (e.g., (1) a reference system for surgical objects, and (2) a 3D coordinate system for X-ray images). Correspondingly, the position and orientation between the object and the coordinate system can be partially or completely described.

[0017] For example, the 2, 3, 4, or 5 degrees of freedom of a surgical object may be determined in the image 3D coordinate system. Based solely on a single 2D projection image, the spatial position of an object in the image 3D coordinate system may be determined, for example, by 5 degrees of freedom. In the context of this disclosure, this indicates “localization” or “imperfect registration,” meaning that the position in the X-ray direction is determined with significantly less precision than its 2D position in the image and its additional 3 degrees of freedom (DOF) of rotation. Furthermore, some components of the 3D orientation may also be localized, but with reduced precision or completely unmeasurable. A cylindrical object may be described via two precise positional degrees of freedom and one precise rotational degree of freedom, with the other degrees of freedom being less precise or even indeterminate.

[0018] Throughout this disclosure, the term “imaging direction” (also called “line of sight direction”) on an object (or region of interest) means the 3D angle at which the X-ray beam passes through a selected point on the object (or region of interest). For example, the imaging direction on a region of interest describes the orientation of the X-ray machine relative to anatomical structures within the region of interest. The imaging direction may be determined as part of the localization of the object (e.g., the anatomical structures of a patient), thereby emphasizing that this is a spatial relationship between the object (e.g., region of interest) and the X-ray image 3D coordinate system.

[0019] Based on a single projection image, it may not be possible to reliably determine the tilt and rotation in the direction of image depth due to the object being too thin to be fully visible in the image (typically always the case with a drill bit) and the inability to accurately determine its imaging depth. In this case, determining 4DOF may be sufficient as an intermediate step to determine 5DOF or 6DOF, based on additional steps such as receiving information about the 3D position of anchor points in a coordinate system, such as the 3D coordinate system of another object of interest, such as the anatomical structure of a patient. This allows for the determination of its 3D position and its tilt relative to other objects.

[0020] In navigation tasks, determining 3DOF (3 degrees of freedom) in translation and 2DOF (2 degrees of freedom) in rotation may be perfectly sufficient. This is especially true when determining the relative 3D position of, for example, a drill bit to the anatomical structure of a patient. In this case, determining the rotational position around the drill axis may be irrelevant.

[0021] 6DOF adequately describes the position and rotation of any stationary object. Some objects, such as Kirschner wire or drill bits, exhibit deformations such as bending and may not be fully described by 6DOF, but may only be described with a larger degree of freedom, for example, 8DOF. In such cases, it may still be sufficient to focus on a subset of degrees of freedom that adequately describes the part of the object of interest. For example, the tip position of a bent drill bit could be described with 3DOF, or the area around the tip could be located with 4DOF.

[0022] The proposed system and method (to be implemented as a computer program product) is configured to determine the position of an object in a 3D coordinate system based on an X-ray image.

[0023] If a portion of the surgical object is depicted in an X-ray image and the geometric shape of a portion of the surgical object is known, incomplete registration may involve determining at least three degrees of freedom of the position of the surgical object in a 3D coordinate system.

[0024] Alternatively or additionally, the reference object may be attached to a bone, at least a portion of the reference object may be depicted in an X-ray image, and the geometric shape of at least a portion of the reference object may be known. In such cases, the incomplete registration may include determining at least three degrees of freedom of the position of the reference object in a 3D coordinate system.

[0025] According to one embodiment, the system may include a robotic device for treating a patient, and a surgical object may be attached to the robotic device. As a result, the system may be configured to control the movement of the robotic device. The robotic device may include a robotic arm configured to hold a surgical object for treating a patient, and the controlled movement of the robotic device may be the movement of the robotic arm together with the surgical object with respect to at least one degree of freedom. Furthermore, the system may include an input device, the input of which defines the movement of the robotic arm. A system including one or more robotic arms that can be controlled by the system and / or a physician allows a physician to operate on a patient remotely by combining automated robotic steering provided by intraoperative navigation with manual robotic steering, while covering all surgical tasks and allowing the surgeon to be outside the radiation field of the imaging device.

[0026] According to one embodiment, the system can include a plurality of cameras, and at least one pose of the cameras is mounted on a robotic arm controlled by the system. Further, the system, i.e., the processing unit, can provide the calculated position of the robotic device, and the system further includes a display for visualizing the calculated position. The calculated position can be visualized before the robotic device is controlled to change its position to the calculated position. In particular, information such as the calculated position can be visualized on an augmented reality function with a display.

[0027] For example, the system can control the operation of a robotic device for soft tissue treatment of a patient, and the soft tissue treatment is controlled or restricted by the system in the case of manual operation of the robotic device. The soft tissue treatment of the patient can be one of removing at least a part of an intervertebral disc, performing an incision, treating the dura mater, treating the spinal cord, treating a nerve, and treating a blood vessel.

[0028] Further, the system can control the operation of a robotic device for bone treatment of a patient, and the bone treatment is controlled or restricted by the system. The treatment of the patient's bone can be one of drilling, resection, repositioning of at least one bone fragment, insertion of an implant, connection of an implant, removal of an implant, insertion of a bone graft, insertion of a cage for bone grafting, insertion of an artificial intervertebral disc, and performing osteotomy.

[0029] Further, it is possible to provide real-time tracking of instruments (no additional tracker or reference attached to the patient's anatomical structure is required) to determine the 3D position of at least one surgical object with respect to the patient's anatomical structure at another point in time. This can be the same even when the patient's anatomical structure (region of interest) is moving.

[0030] Aspects of the present disclosure may be applicable to open surgery, minimally invasive surgery, endoscopic surgery, and percutaneous spinal surgery in the treatment of degeneration, deformity, tumors, trauma, or infections, including, but not limited to, cervical and thoracolumbar discectomy surgery, decompression procedures, motion preservation surgery, stabilization procedures such as vertebroplasty and kyphoplasty, spinal fixation procedures such as ACDF, TLIF, PLIF, LLIF, OLIF, XLIF, open, minimally invasive, or percutaneous pedicle screws, tumor debulking surgery, or vertebrectomy.

[0031] The surgical object may be a surgical instrument or device such as a drill, needle, or chisel, or it may be an implant such as a pedicle screw, nail, or plate. The surgical object has a known geometric shape described by a 3D model. The 3D model can be a detailed abstraction that describes, for example, the surface and density of the object, or it can be a less detailed abstraction that includes, for example, length, diameter, or a partial surface description.

[0032] The surgical object can also have a point designated as an "anchor point", the position of which may be known in the 3D model of the surgical object and can be identified in an X-ray image. The anchor point may be, for example, the tip of a drill. In the case of a chisel, the anchor point may be at one edge or the midpoint of the chisel blade. The anchor point can be more abstract, such as the center of a cylindrical surgical instrument that is not necessarily part of the object, or an axis passing through the center of the object. [[ID= 10]]

[0033] A 3D dataset describing a portion of a patient's anatomical structure may be generated by a computed tomography (CT) scan or some other type of 3D X-ray scan, which may be acquired preoperatively or intraoperatively. The 3D dataset includes a region of interest, which must be sufficiently rigid. The region of interest may be, for example, a volume containing a bone fragment, a volume containing a specific vertebra, or a volume containing two adjacent vertebrae, provided they are assumed to be sufficiently rigid. The region of interest may also be the entire 3D dataset. Since the 3D dataset accurately describes the patient's anatomical structure, at least within the region of interest, a one-to-one correspondence between points in the physical world and points in the 3D dataset can be established, at least within the region of interest. Thus, the 3D dataset can define a 3D coordinate system in physical 3D space. This can be used to determine the 3D position of a surgical object in this 3D coordinate system, or, in other words, to determine the 3D position of a surgical object (in the physical world) relative to the region of interest within the patient's anatomical structure.

[0034] In preoperative or intraoperative planning, the 3D dataset of the patient's anatomical structures and target points within the 3D dataset can be determined. Target points may be determined preoperatively or intraoperatively either manually by the surgeon, automatically by the system, or semi-automatically by the surgeon with some automated support from the system. Thus, the location of the target points in the 3D coordinate system defined by the 3D dataset is known. Target points may be selected such that the anchor points of surgical objects can be placed at (or near) the physical location of the patient's target points, which can be achieved, for example, by selecting target points on the bone surface.

[0035] The target point may be the starting point of the target path, and its ending point is called the target ending point. The target path lies within the region of interest, and its position and orientation within the 3D dataset can be determined in preoperative or intraoperative planning either manually by the surgeon, automatically by the system, or semi-automatically by the surgeon with some automated support from the system. Therefore, if the target path is available, its position and orientation in the 3D coordinate system are known. For example, when treating a vertebra, the target path may correspond to the planned drill trajectory (typically a straight line), and the target point is located on the surface of the vertebra.

[0036] This disclosure teaches a method for determining the 3D position of at least one surgical object in a 3D coordinate system defined by a 3D dataset describing the anatomical structures of a patient. By doing so, the relative 3D position of the surgical object to a region of interest in the patient's anatomical structures is determined. Thus, if the region of interest includes a planned target path, the relative 3D position of the surgical object to the target path within the patient's anatomical structures can also be determined. It may also be aimed at providing instructions for aligning the surgical object with the target path (if available) for, for example, drilling along the target path.

[0037] It should be noted that some surgical objects, such as drill bits, have rotational symmetry around their axis. In the case of such objects, rotation around the object's axis is irrelevant, so it is sufficient to determine only the five degrees of freedom of their 3D position relative to the patient's anatomical structure. For example, in the case of a drill bit, it is sufficient to determine only the 3D position of the drill tip and the 3D orientation of the drill axis.

[0038] In the context of this disclosure, the rotation of a surgical object about its axis may be independent of the 3D spatial relationship of the surgical object to the anatomical structure. In other words, the 3D position is considered to be determined regardless of whether the actual rotation angle of the surgical object about its axis of rotation is precisely determined or not.

[0039] Neither a target point nor a target path is always necessary. It may even be possible to work using only anchor points (for example, the tip of a drill, or a geometric anchor configuration). In this case, the surgeon can determine during the surgery whether the 3D position and orientation of the surgical object relative to the patient's anatomical structure is satisfactory, and if not, how to correct it.

[0040] It is emphasized that 3D datasets do not necessarily need to be segmented. In particular, it is not necessary to identify bone surfaces within a 3D dataset (except for selecting target points on the bone surface). Furthermore, it is not necessary to know, or explicitly determine, the exact location and shape of the region of interest within the 3D dataset. For example, when treating a vertebra, a CT scan may include this vertebra and several adjacent vertebrae (and surrounding soft tissue). Initially, it is not necessary to identify the vertebra within the 3D dataset. A rough approximation of the region of interest within the 3D dataset can be derived from the target points. Anchor points define the 2D region of interest in the X-ray image. For subsequent calculations, data points within the 2D region of interest can be weighted, with points further away from the anchor points receiving lower weightings.

[0041] As described above, the localization or registration of a 3D region of interest depicted in an X-ray image can be determined by calculating a number of digitally reconstructed X-ray images (DRRs) for localization from a 3D dataset. Alternatively, a model of the imaged structure may be used to determine the localization. Localization may also be determined based on position sensors and / or motion sensors provided on the imaging device. Localization may also be determined by starting from a previous registration of the 3D data and the 3D image coordinate system, tracking the imaging device with an arbitrary tracking device such as a mixed reality (XR) device, and adding the detected relative movement of the imaging device to the previous localization of the previous registration.

[0042] When determining localization by acquiring a DRR that best matches the X-ray image, it may be possible to use a numerical optimizer to find the optimal DRR. The DRR does not need to be limited to the region of interest. When evaluating the best match between the DRR and the X-ray, the region of interest can be emphasized (e.g., by appropriate weighting). If the acquired X-ray image is distorted (i.e., acquired using an image intensifier rather than a flat panel receiver), the distortion may be stronger as the distance between points increases, so it may be necessary to use a smaller area in the X-ray image for DRR matching (or emphasize points closer to the anchor point). Since distortion usually has a stronger effect on areas further away from the central X-ray beam, it may be useful to ensure that the anchor point is close to the center of the X-ray image.

[0043] As described in detail in U.S. Patent Application Publication No. 2021 / 0248779, the 3D position of thin surgical objects, such as drill bits, is not always uniquely determined based on a single X-ray image. Therefore, generally, when only one X-ray image is available, there may be ambiguity in the relative 3D spatial relationship between the surgical object and the patient's anatomical structure. This disclosure teaches how to resolve such ambiguity by first determining the 3D position of anchor points in a 3D coordinate system defined by a 3D dataset. Various methods for achieving this are disclosed.

[0044] If no target point is used, the anchor point may be located on the bone surface. Based on a single X-ray image, the 2D position of the anchor point can be determined within the X-ray image, and the direction of X-ray image acquisition over the region of interest can be determined from DRR matching. In 3D space, the line is determined by the point on the anchor point in the virtual projection onto the X-ray image. Since the anchor point, being a geometric anchor, is on the bone surface, the position of the anchor point in 3D can be found by determining the point on the line on the bone surface. It is sufficient to determine the position on the bone surface along the line. Segmentation of the entire bone surface is not necessary. "On the bone surface" or "on the bone surface" is also understood to include the fact that it is sufficient for the anchor point to be in the vicinity of the bone surface, e.g., less than 2.5 mm or less than 1.5 mm from the bone surface. Alternatively, it is possible to know exactly how far the anchor morphism is from the bone surface. In this case, this distance can be taken into consideration.

[0045] Another method involves establishing a correspondence between anchor points (on the surgical object) and target points (which may be geometrical aspects of a 3D dataset). This is done by virtually projecting the target points into the X-ray image, for example, by utilizing known imaging directions on the region of interest determined by DRR matching.

[0046] If the position of an anchor point matches the position of a target point in an X-ray image, it can be assumed that the 3D position of the anchor point matches the 3D position of the target point in physical space. This is because the target point is selected so that the anchor point of the surgical instrument can be positioned on the target point (in physical space). Of the three coordinates of the 3D position, two degrees of freedom are determined by the X-ray image, and the third degree of freedom is determined by prior information that the target point and the anchor point are on or at a defined distance from the bone surface.

[0047] For example, if the anchor point is not on the bone surface because drilling into the pedicle has already begun, yet another method for determining the 3D position of the anchor point in a 3D coordinate system defined by the 3D dataset is to use additional X-ray images from a different imaging direction. These additional X-ray images may be acquired at an earlier point in time, or simultaneously with the first X-ray image at the first point in time (e.g., if a G-arm is used), or at a later point in time (e.g., if a robot capable of holding the surgical object stationary is used). The additional X-ray images depict the surgical object and at least a portion of the region of interest within the patient's anatomical structure. The 2D position of the anchor point in the additional X-ray images is determined. The imaging direction of the additional X-ray images onto the region of interest is determined, thereby allowing the additional X-ray images to be registered to the first X-ray images. This method assumes that there was no movement of the anchor point relative to the anatomical structure between the generation of the two X-ray images. The advantage of this method is that preoperative planning (e.g., drill trajectory and / or target point definition) is unnecessary. Furthermore, if the anchor point (for example, the tip of a drill) is located below the bone surface, the possibility of the surgical instrument accidentally slipping is minimized.

[0048] An anchor point may also be a geometric aspect of a surgical instrument, such as the tip and central axis of a drill bit. 3D position estimation can be achieved if the anchor point can be partially localized in the X-ray image at a first time point, or at a later time point. For example, the central axis of a drill bit may be identifiable in the X-ray image at the first time point, while only the tip of the drill bit may be identifiable in the X-ray image at a later time point. Assuming the drill tip did not move, it is still sufficient to estimate the 3D position of the anchor point in a 3D coordinate system. Given the direction of X-ray imaging onto the drill bit at both the first and later time points, the detected tip and central axis in the image yield geometric aspects that can be transformed from the image 3D coordinate system to a 3D coordinate system. The tip of the drill bit at the later time point describes a line in the 3D coordinate system, and the central axis at the first time point describes a plane in the 3D coordinate system. The intersection of the line and the plane provides an estimate of the 3D position of the anchor point. In this case, the 3D orientation of the central axis of the anchor point is only partially estimated because, in order to estimate it more accurately, the central axis needs to be identified in both X-ray images.

[0049] The estimation of the 3D position of an anchor configuration, which may be geometric, can be achieved in a similar manner even when the surgical instrument is in motion. In the example of a drill bit, the anchor configuration of the surgical instrument may be its central axis and its tip position. The drill bit may be positioned in the 3D coordinate system of an anatomical structure, for example, within bone. A first X-ray may be acquired at a first time point, and the drill bit may be positioned such that its tip is determined in 2D coordinates (at least 2DOF). The first X-ray image corresponds to a 3D image coordinate system, and the tip positioning corresponds to an indeterminate line between the X-ray source and the projected tip in the 2D image plane. At a second time point, another X-ray may be acquired, in which case the registration between the 3D image coordinate system and the 3D coordinate system of the anatomical structure is also received or determined.

[0050] At the second time point, the drill bit may have been inserted further, so that the tip is not in the same 3D position (relative to the anatomical structural coordinate system) compared to the first time point, although the tip at the first time point is still near the axis of the drill bit at the second time point. The drill bit may be positioned in the second X-ray image such that its tip and axis are determined in 2D (at least 3DOF), with the tip corresponding to an indeterminate line in the second 3D image coordinate system and the axis corresponding to a plane in the second 3D coordinate system.

[0051] Using registration between coordinate systems, uncertain lines from the first 3D image coordinate system, and uncertain lines and planes from the second 3D image coordinate system, can be transformed into the 3D coordinate system of the anatomical structure. In this 3D coordinate system, the intersection points of the uncertain planes from the second X-ray image and the uncertain lines from the first X-ray image become 3D points on the axis of the drill bit at the second time point.

[0052] To fully determine the 3D positions of the drill tip and drill axis, it is necessary to constrain another degree of freedom, which may have several origins. For example, (i) if the direction remains the same between the first and second X-rays, the angle may be determined by the drill bit from the first X-ray (i.e., at least 3DOF), (ii) the positioning of the drill bit in the second X-ray can provide a more detailed estimate of rotation (i.e., at least 4DOF), and (iii) there may be reasonable assumptions about a fourth degree of freedom. This constraint allows the axis of the drill bit at the second time point to be determined. Given the axis of the drill bit, the tip point of the drill bit at the second time point can be determined as the intersection of the axis of the drill bit and its indeterminate line from the second X-ray image. Given the tip position and axial position (i.e., the 3D position of the anchor configuration) of the drill bit, 5DOF at the second time point is determined. The rotation of the drill bit around its spindle may also be determined as part of the positioning in the second X-ray image.

[0053] The geometric properties of an object are descriptive attributes. Examples of geometric properties include a subset of the surface of a drill bit, a feature point at the tip, the centroid of a bone surface, the projected trajectory of a drill bit, and the curved central axis of a curved cylindrical surgical instrument. Depending on the geometric property, less than 6DOF may be required to fully describe the geometric property in relation to a 3D coordinate system. The tip of a drill bit can be fully described by its position, which has 3DOF, and only 5DOF is needed for the line. In the case of a curved cylindrical surgical instrument, the central axis may have more than 6DOF to fully describe it, but it may be sufficient to reduce the set of parameters and only partially estimate its properties.

[0054] Typically, a 3D dataset (e.g., a CT scan) contains scaling information (i.e., size information) of the anatomical structures described in the dataset. Furthermore, the scale of the scene depicted in the X-ray can also be provided by depicted surgical objects whose precise geometric shapes are known, or by other surgical objects with known geometric shapes that can be depicted in the X-ray image (e.g., already implanted pedicle screws). This redundancy regarding scale information can be used to distinguish different surgical objects in the X-ray image that have the same geometric shape but differ in size (e.g., drills with different diameters) by comparing the scale of the 3D data with the size of the surgical objects. If the acquired X-ray image is distorted, the same information (i.e., the 3D dataset, the 3D model of the surgical object, or another surgical object with a known geometric shape) can also be used to correct the distortion.

[0055] In the case of a distortion-resistant X-ray imaging device (i.e., an image intensifier), the system can account for distortion by, for example, using parameters that describe distortion. For example, the system can use one parameter to describe pincushion distortion and one parameter for so-called S-type distortion. Distortion can be accounted for by, for example, applying a distortion model to distort multiple virtually rendered images, thereby comparing the distorted actual X-ray image with the distorted virtual X-ray image. Alternatively, the system can use distortion parameters to eliminate distortion in the actual X-ray image, and thus compare the distortion-free actual X-ray image with multiple distortion-free virtual images. If the distortion parameters are unknown, the system can use a fixed set of parameters for distortion (e.g., the average distortion evaluated across multiple X-ray imaging devices). Alternatively, the system can estimate distortion by comparing two or more actual X-ray images acquired by moving the X-ray imaging device, and as a result, it can be assumed that the distortion is constant for all acquired X-ray images. Since these images can depict at least one identical part of an anatomical structure (i.e., the same part of the anatomical structure in a different part of the image due to movement of the imaging device), 2D-2D matching can be enhanced not only to estimate image transformation parameters but also to estimate distortion parameters.

[0056] After the first step of the registration procedure, i.e., after acquiring an X-ray image in which the anatomical structure has been successfully localized (5DoF) in the X-ray image coordinate system or 3D coordinate system, if the position of the anchor point does not match the position of the target point, the system may provide navigation information by moving the surgical instrument so that the anchor point approaches the target point, and then, for example, a new X-ray image may be acquired to provide complete registration, or additionally or alternatively, the object having the anchor point may already be registered in either the image coordinate system or a 3D coordinate system whose transformation to the image coordinate system is at least schematically known, and is navigated / tracked in real time, and when the anchor point becomes the target point, this information is provided to the system (e.g., by applying pressure to the sensor or by user input) to provide complete registration.

[0057] After acquiring the first X-ray image and locating the anatomical structures, the system can check all target points visible in the X-ray image if there is ambiguity, i.e., if two surface points are close to each other (e.g., 3D distance less than 15 mm) and cannot be distinguished, such that two surface points are depicted in similar 2D coordinates in the X-ray image. The system can mark and / or exclude target points with ambiguity so that they cannot be used in a complete registration procedure, for example, based on the proximity of anchor points to target points. Additionally or alternatively, the system can calculate the required imaging orientation to eliminate ambiguity, or at least eliminate ambiguity of target points that are most likely to be used as the next step, and provide instructions on how to reposition the imaging device to acquire the X-ray image in the optimized imaging orientation.

[0058] The target point may be on the surface of the bone, or, for example, 3 mm below the surface. If the target point is, for example, 3 mm below the bone surface, the user or, for example, the robot must ensure that the complete registration or the second step of registration is also, for example, 2-4 mm below the surface. In other words, to ensure accurate registration, it is necessary to ensure that the target point is sufficiently close in 3D.

[0059] If the anchor point is already located near the target point in the X-ray image, and a local model of the bone surface near the target point is available, the 3D position of the anchor point can be obtained from the local model relative to the 2D position of the anchor point detected in the X-ray image. This is sufficient to determine the 3D relative spatial relationship between the surgical object and the patient's anatomical structure. Such a local model of the bone surface may be, for example, the average shape of the bone, a first-order approximation of the bone shape, or a local segmentation of the bone surface near the target point. The correct model may be selected based on the labeling of the 3D dataset (e.g., classification of vertebrae), and this may be done automatically, semi-automatically, or manually. The size of the target point's neighborhood can be fitted based on the local model; for example, the smaller the variation in the local model, the larger the range (vicinity) that can be treated as the target point's neighborhood. Of course, if segmentation of the 3D dataset (i.e., a model of the entire bone surface) is available, this can be taken into account to improve accuracy.

[0060] The 3D position of an anchor configuration is sufficiently accurate in terms of adjustable dimensions, for example, when navigating a surgical instrument to a target point, but there may be significant errors in some dimensions. When navigating an anchor configuration to a target point, navigation information may be constrained to 2D translation (e.g., bone surface), but the 3D translation can only be estimated with greater error. This is particularly useful when 2D navigation can be estimated based on an X-ray image, in which case the 2D navigation direction is not perpendicular to the projection plane of the X-ray image.

[0061] Alternatively or additionally, the localization of the anchor configuration can utilize a target point to constrain its 3D coordinates. For example, if the drill tip can be estimated in 2D within the X-ray image, while the target point is estimated or known in the 3D image coordinate system, then the position on the uncertain line (i.e., the line from the X-ray source to the projected anchor configuration in the image plane) can be assumed to be close to the target point.

[0062] Alternatively or additionally, the system may include means for tracking the anchor morphology from the time of the acquired X-ray to any subsequent point in time, utilizing the estimated position of the anchor morphology in a 3D image coordinate system relative to the target point. Tracking may be performed via external sensors, via a camera, or from instructions or sensors of a surgical robot. Based on this tracking, the direction and distance to the target point may be updated and provided to the surgeon or robot in real time (or near real time).

[0063] It will be understood that the target point needs to be linked to an estimate of the anchor configuration in order to provide navigation instructions. For example, the target point may be derived from a 3D dataset, and the 3D coordinate system of the 3D dataset is registered to a 3D image coordinate system so that the target point can be mapped to the 3D image coordinate system.

[0064] As mentioned above, it is not necessary to know the exact location and shape of the 3D region of interest within the 3D dataset. Furthermore, it is not necessary to establish a precise correspondence between the 2D region of interest in the X-ray image and the 3D region of interest in the 3D dataset. However, there may be cases where several possible 3D regions of interest exist. For example, in spinal surgery, multiple vertebrae are treated, so multiple 3D regions of interest may exist. In such cases, a human surgeon can manually identify which 3D region of interest should be treated. It may also be possible to automatically select the relevant 3D region of interest corresponding to the anchor point indicated by the surgical instrument by performing DRR matching for each possible 3D region of interest and selecting such best match.

[0065] The region of interest is an approximation of an area (or volume) that can be assumed to be sufficiently rigid so that the 3D dataset describes the patient's anatomical structures within the region of interest with sufficient accuracy. For example, because the spine is flexible, individual vertebrae can move relative to one another. If the patient moves after the 3D dataset is acquired, some relative movement of individual vertebrae must be expected. This means that the region that can be assumed to be rigid may be larger in intraoperative 3D X-rays (without subsequent patient movement) than in preoperative CT scans (with subsequent patient movement), assuming no significant movement due to respiration.

[0066] The 3D dataset may be input into the system or derived by the system. For example, the system may estimate 3D data based on X-rays using a statistical model of relevant parts of an anatomical structure. As an example, in the context of spinal surgery, the statistical model (e.g., an active shape model) may describe the surface or density of one or more vertebrae. The 3D dataset may also be updated or extended with further X-ray images to improve accuracy and reliability.

[0067] To improve the accuracy of the derived 3D dataset, the system can utilize the geometric configuration of a surgical object by registering its configuration across two or more images. For example, a drill bit may be positioned on bone in the first and second X-ray images, with the drill axis changing between images, but the tip remaining the same. Similarly, a screw may be identified in both X-ray images to limit the possible viewing directions of the X-ray images. This may be used to determine or limit the registration of the 3D image coordinate system, thus improving its accuracy.

[0068] Alternatively or additionally, prior knowledge regarding the geometric configuration of surgical instruments can be utilized. For example, if the drill tip is positioned on the bone surface, the modeled bone surface needs to be close to the detected drill tip position. Using drill tip positioning can constrain the system when estimating 3D data, thus improving the accuracy of the estimation.

[0069] The methods taught herein can also complement existing navigation techniques. Another aspect may involve continuously incorporating information from intraoperative X-rays. The systems and methods disclosed herein do not require a navigation camera or other sensor, but can nevertheless be combined with a navigation camera or other sensor (e.g., a sensor mounted on a robot). Information from X-ray images and information from cameras or other sensors can be combined to improve the accuracy of determining relative spatial position and orientation, or to resolve any remaining ambiguities (which may be due to, for example, occlusion). If the instrument is held by a robot or robotic arm, information provided by the robot or robotic arm itself (e.g., about the movements it has performed) can also be considered.

[0070] The methods taught in this disclosure can also be combined with augmented reality devices such as head-mounted augmented reality glasses or mixed reality glasses. For example, by tracking a power tool with a head-mounted augmented reality device, it is possible to determine changes in the 3D position and 3D orientation of a drill bit inserted into the power tool between two time points. Thus, if the 3D position of a surgical object (such as a drill bit) relative to an anatomical structure is known at a first time point, the 3D position of the surgical object relative to the anatomical structure can be determined at a second time point by tracking the power tool with an augmented reality device. If only the tilt of the surgical object is known to have changed between the two time points (for example, the power tool was only tilted, but the drill tip remained in place), this knowledge can be used by the system.

[0071] It should be noted that information regarding the change in the 3D position of the surgical object between two points in time can also be provided by the robotic arm holding the surgical object, since the robotic arm can typically track the movement of the surgical object in 3D space.

[0072] Applications to which this disclosure may be applied include all types of bone drilling, such as screw insertion into the pedicle, wedge osteotomy, screwing into the sacroiliac joint, screwing two vertebrae together, screwing through the pelvis, screwing through the acetabulum, or drilling of the cruciate ligament. The disclosed teachings may also be used, for example, for drilling, reaming, milling, chiseling, sawing, resection, and implant positioning, and thus can also support, for example, osteotomy, laminectomy, tumor resection, and total hip replacement.

[0073] It will be understood that a system may include processing units, and methods may be implemented as computer program products that can be executed on those processing units.

[0074] According to one embodiment, the system and method can cause an imaging device to generate an X-ray image or some type of 3D X-ray scan (e.g., a CT scan). Additionally or alternatively, the system and method can control the imaging device to move to a new position in order to generate an X-ray image from a different imaging direction. Such a different imaging direction may be a suitable imaging direction proposed by the system. The current imaging direction may be determined based on an internal motion sensor and / or position sensor of the imaging device.

[0075] It should be noted that the image data of the processed X-ray images can be received directly from an imaging device, such as a C-arm, G-arm, or two-plane 2D X-ray device, or from a database. A two-plane 2D X-ray device has two X-ray sources and two receivers offset at any angle.

[0076] According to one embodiment, objects in an X-ray image can be automatically classified and identified within the X-ray projection image. However, objects may also be manually classified and / or identified within the X-ray projection image. Such classification or identification may be supported by the device by automatically referencing structures recognized by the device.

[0077] It should be noted that a processing unit can be implemented by a single processor that performs all steps of the process, or by a group or multiple processors that do not need to be located in the same place. For example, in cloud computing, processors can be placed anywhere. For instance, a processing unit could be divided into a first subprocessor that controls user interaction, including a monitor for visualizing results, and a second subprocessor (which may be located elsewhere) that performs all the calculations. The first or other subprocessor could also control, for example, the movement of the C-arm or G-arm of an X-ray imaging device.

[0078] According to one embodiment, the device may further include storage means that provide, for example, a database for storing X-ray images. Such storage means may also be provided in a network to which the system may be connected, and it will be understood that data related to a neural network may be received through that network. Furthermore, the device may include an imaging unit for generating at least one 2D X-ray image, and the imaging unit may be capable of generating images from different directions.

[0079] According to one embodiment, the system may include a device for providing information to a user, the information including at least one of a group consisting of X-ray images and instructions relating to procedural steps. It will be understood that such a device may be a monitor or augmented reality device for visualizing the information, or a speaker for providing the information acoustically. The device may further include input means for manually determining or selecting the location or part of an anatomical structure in an X-ray image, such as a bone contour, for measuring distance in the image, for example. Such input means may be a computer keyboard, computer mouse, or touchscreen for controlling a pointing device, such as a cursor on a monitor screen, and these may be included in the device. The device may also include a camera or scanner for reading package labels or identifying surgical objects. The camera may also allow the user to communicate visually with the device by gestures or facial expressions, for example, by virtually touching the device displayed by virtual reality. The device may also include a microphone and / or speaker for acoustic communication with the user.

[0080] According to one embodiment, the system may also comprise an augmented reality device. Since the augmented reality device covers all devices on the augmented reality spectrum, it will be understood to be a superset of virtual reality, augmented reality, and mixed reality devices. This represents all devices that a user can wear and that (1) can alter the user's perception through any sense, including sight, hearing, smell, taste, and touch, and (2) can track its environment through various sensor information, including cameras, depth sensors, accelerometers, and magnetometers. These sensors can track an environment that includes an imaging device and / or at least one object, the object being an anatomical model, surgical instrument, or implant, etc. It should be noted that the tracking information makes it possible to distinguish between objects and / or devices. For example, the augmented reality device may be head-mounted, allowing the user to see the environment, but may also comprise a pair of glasses configured to visualize information or virtual representations of objects within the user's field of view.

[0081] It should be noted that all references to C-arm movement in this disclosure always refer to relative repositioning between the C-arm and the patient. Therefore, any C-arm translation or rotation can generally be replaced by a corresponding translation or rotation of the patient / OR table, or a combination of C-arm translation / rotation and patient / table translation / rotation. This may be particularly relevant when dealing with limbs, as it may actually be easier to move the patient's limbs than to move the C-arm. It should be noted that required patient movement generally differs from C-arm movement, and in particular, patient translation is typically not required if the target structure is already in the desired position in the X-ray image. The system can calculate C-arm adjustments and / or patient adjustments. Furthermore, it should be noted that all references to the C-arm may also apply to the G-arm, O-arm, etc.

[0082] The methods and techniques disclosed herein may be used in systems supporting a human user or surgeon, or in systems in which some or all of the steps are performed by a robot. The robot performing the steps may mean that the robot performs the steps automatically based on system output or manually based on the surgeon's manual operation. Thus, references to “user” or “surgeon” in this patent disclosure may refer to a human user as well as a robotic surgeon, a mechanical support device, or similar device. Similarly, where it is mentioned that instructions are given on how to adjust the C-arm, it should be understood that such adjustments may be performed without human intervention, i.e., automatically, by the robotic C-arm, by the robotic table, or by the OR staff using some kind of automated support. It should be noted that because robotic surgeons and / or robotic C-arms can operate with greater precision than humans, repetitive procedures may require fewer repetitions, and more complex instructions (e.g., combinations of multiple repetitive steps) may be performed. A key difference between robotic surgeons and human surgeons is the fact that robots can keep instruments completely still between the acquisition of two X-ray images.

[0083] If the procedure is performed entirely by a robot, the surgeon (who supervises and potentially / partially operates the robot) can be positioned far enough away from the imaging device or behind a wall, and therefore does not need to wear additional radiation protection.

[0084] When robot-assisted procedures are combined with HMD / augmented reality-assisted procedures, it can be beneficial to display the future position of the robotic arm and / or the surgical instruments held by the robotic arm in augmented reality. Therefore, if the robotic arm is moving precisely to a position that has already been virtually displayed, this provides reliability and further safety. For example, in a scenario where the system virtually displays where the robotic arm or instrument should be placed, and the robotic arm does not move precisely to this position, the user can recognize the problem and, for example, trigger a new registration of the robot or remove any mechanical obstacles that may be preventing the robot from moving to the desired position.

[0085] The computer program product may preferably be loaded into the random access memory of a data processor. Therefore, a data processor or processing unit of a system according to one embodiment may be equipped to perform at least a portion of the described process. Furthermore, this disclosure may relate to computer-readable media, such as a CD-ROM, capable of storing the disclosed computer program product. However, the computer program product may also be provided via a network such as the World Wide Web and can be downloaded from such a network into the random access memory of a data processor. Furthermore, the computer program product may also be executed on a cloud-based processor, with the results presented via the network.

[0086] It should be noted that prior information regarding surgical objects (e.g., drill bit size and type) can be obtained before or during surgery by simply scanning the packaging (e.g., barcode) or reading the information printed on the surgical object itself.

[0087] As is evident from the above description, the primary aspect is the processing of X-ray image data to enable the automatic interpretation of visible objects. The methods described herein should be understood as methods to assist in the surgical treatment of patients. Consequently, according to one embodiment, the method may not include the step of treating an animal or human body by surgery.

[0088] It will be understood that the steps of the methods described herein, in particular the steps of the methods described in relation to the workflow of embodiments in which some are visualized in the figures, are major steps, and these major steps may be differentiated or divided into several substeps. Furthermore, there may be additional substeps between these major steps. It will also be understood that the invention may consist of only a part of the whole method, i.e., steps may be omitted or summarized.

[0089] It should be noted that the embodiments are described with reference to different subject matter. In particular, some embodiments are described with reference to method-type claims (computer program products), and other embodiments are described with reference to apparatus-type claims (systems / devices). However, those skilled in the art will understand from the above and below descriptions that, unless otherwise specified, any combination of features belonging to one type of subject matter, as well as any combination of features relating to different subject matters, is disclosed in this application.

[0090] The embodiments, as well as further embodiments, features, and advantages of the present invention as defined above, can also be derived from the examples of embodiments described below, and will be explained with reference to the examples of embodiments shown in the figures, but the present invention is not limited thereto.

[0091] The tip of an object, such as the tip of a sleeve, may or may not be depicted in a 2D X-ray image because it may be obscured by other parts of the same object or by other objects. Nevertheless, it is possible to accurately detect a sufficient contour and potentially use the inner appearance of the sleeve in the X-ray image, and thus it may be possible to use this information to determine its relative 3D position and orientation to anatomical structures. This information can then be used as a basis for tracking the object, for example, by a camera and / or 3D sensor on a head-mounted display. By tracking the 3D position of the sleeve, the drill trajectory can be determined and / or observed even if the drilling or drilling machine itself is not tracked at all. This may be advantageous because tracking the drilling machine to determine the actual drill trajectory can result in inaccuracies or even significant deviations between the drill trajectory and the drilling machine's trajectory, for example, because a portion of the drill bit outside the bone (and outside or without the sleeve) may bend significantly, especially with long drill bits.

[0092] It may be advantageous to combine sleeve tracking to determine the drill trajectory (i.e., determination of spatial relationships at additional points in time, e.g., real-time tracking) with drill machine tracking to determine, for example, the drilling depth of the drill bit within the bone.

[0093] It may be advantageous to track the relative position of the drilling machine to the sleeve. This tracking may suffice if performed only in one dimension, for example, by the distance of the (calculated) position of the drill tip to the position of the sleeve tip in the direction of the drill trajectory.

[0094] To ensure precise tracking of the sleeve, it may be molded so that a sufficient portion of it is not obstructed by other objects such as instruments, drills, or the surgeon's hand or arm.

[0095] To ensure the precise detection and determination of the 3D position of the sleeve relative to an anatomical structure, for example, based on 2D X-ray image processing, it may be advantageous to have sufficient radiopaqueness of the sleeve, or to have a sleeve-holding arm that is securely attached to the sleeve over a sufficient volume.

[0096] To enable precise detection and image processing of a sufficient portion of the drill bit, it may be advantageous for the sleeve portion that actually covers the drill bit to be semi-transparent.

[0097] After incision, the procedure can begin with positioning / navigating the drill sleeve. As soon as the drill sleeve is positioned on the surface of the anatomical structure at the desired / planned entry point, a drill bit attached to an insertion instrument (e.g., a power tool) can be inserted into the drill sleeve until it contacts the bone surface. A real-time tracking system (e.g., a head-mounted augmented reality device or robotic arm) can track the insertion instrument and detect when it is on the surface. This detection can be made more precise by, for example, rotating the insertion instrument as soon as the tip of the drill bit reaches the surface. The drill sleeve can always remain at an anchor point on the surface of the anatomical structure and move with the patient's respiratory movements. Any change in the angle of the drill sleeve can be determined by a tracking system, which may be provided, for example, by a head-mounted display and its camera and / or 3D sensory devices. Once drilling begins, tracking of the insertion instrument and sleeve can enable updating of navigation information regarding the 3D position of the drill bit within the bone / patient's anatomical structure.

[0098] In summary, a device can be provided for measuring drill depth in moving anatomical structures. The device may comprise a tracking system for tracking a surgical instrument and a processing unit for processing information received from the tracking system, the surgical instrument comprising at least a drill sleeve and a drill bit in a drilling machine, the tracking system tracking the 3D position of the drill sleeve as it moves with the anatomical structure, and further tracking the 3D position of the drill bit as it moves within the drill sleeve relative to the anatomical structure.

[0099] Based on the tracked 3D positions of the drill sleeve and the drill bit at multiple points in time, the device's processing unit may be configured to determine the 3D position of the drill sleeve relative to the 3D position of the drill bit at each of the multiple points in time, and as a result, may be configured to determine the maximum distance of the distal tip of the drill bit from the drill sleeve.

[0100] The maximum distance of the drill tip from the drill sleeve may correspond to the maximum drilling depth of the drill bit into the bone when the drill sleeve is in contact with the outer surface of the bone. [Brief explanation of the drawing]

[0101] [Figure 1] This diagram shows an exemplary workflow for performing a two-step registration. [Figure 2] This figure shows one embodiment of the system according to the present disclosure. [Figure 3] This is a 2D X-ray projection image depicting the spine from the anterior-posterior direction. [Figure 4] This figure shows a slice from the same 3D scan of the spine as Figure 3, depicting a sagittal section. [Figure 5] This figure shows a slice from the same 3D scan as Figure 4, depicting the axial cross-section. [Figure 6]This figure shows an exemplary workflow for performing pedicle screw drilling according to the plan. [Figure 7] This figure shows a short, exemplary workflow for drilling pedicle screws without changing the imaging direction, assuming knowledge of the bone surface. [Figure 8] This figure shows an exemplary workflow for performing pedicle screw drilling based on further X-ray images with different imaging directions. [Figure 9] This diagram shows an example workflow for performing 2D-2D matching. [Modes for carrying out the invention]

[0102] The present disclosure will be described in detail here with reference to the drawings, but the disclosure is made in relation to exemplary embodiments and is not limited to the specific embodiments shown in the drawings.

[0103] This disclosure teaches a system and method for determining the 3D position and orientation of a surgical object relative to a portion of a patient's anatomical structure, called a region of interest, based on a single X-ray image, which is described by a 3D dataset such as a CT scan or any other type of 3D X-ray scan. Without additional information, the 3D pose (i.e., 3D position and orientation) of a thin surgical object, such as a drill, cannot necessarily be uniquely determined based on a single X-ray image.

[0104] The system can also use non-anatomical objects of known geometric shapes as references ideally fixed to anatomical structures to utilize the 3D dataset, by determining the spatial relationship between the 3D dataset and any tracked object in the reference coordinate system. For X-ray images of a region of interest represented by the 3D dataset, and X-ray images that also depict the reference object, the system can orient the 3D dataset relative to the reference object's coordinate system. The accuracy of this orientation is limited; for example, only 5DOF can be accurately identified. A tracked object (e.g., a robotic arm or positioning probe) can be tracked relative to the reference object's coordinate system (e.g., by a head-mounted augmented reality device, such as those used with typical intraoperative navigation systems, or by an optical real-time tracking system, such as one with a stereo camera system) and move to a known region within the 3D dataset. For example, the known region could be a planned drill entry / target point on a bone surface, and the tracked object could be a tracked drill. Once the drill is positioned on the bone surface and, for example, rotates around its tip, the system can detect this rotation and trigger an update to the previous registration / orientation. Alternatively, any other movement, or holding it in place for several seconds, or additional user input to the system can also be applied as a trigger. The update provides a precise determination of the missing sixth DoF of registration, and therefore provides a complete determination of the 3D position and 3D rotation of the 3D dataset relative to the reference object coordinate system. Thus, without additional X-ray images, the update provides a complete registration of the region of interest of the anatomical structure, and therefore also enables accurate real-time tracking of the region of interest relative to any other instrument that is tracked and registered in the reference object coordinate system.

[0105] A crucial part of updating (incomplete) registration is positioning the registered and tracked object near a desired location that constrains the positioning of the 3D dataset relative to the reference object coordinate system. The system can provide guidance or automate this step. For example, the 3D dataset may enable automatic identification of a hard bone surface where the tracked object may be placed. The limited precision of positioning is sufficient to guide the surgeon or robot to this bone surface by known dimensions, and the robot or surgeon moves the tracked object in the direction of limited precision until bone pressure stops the tracked object.

[0106] Another technique for achieving complete 6DoF registration, or the second step of registration, is to move the tracked and registered object to a characteristic point in the 3D dataset. For example, a positioning probe can be moved to such a point on a bone surface and triggered. The trigger is any signal to the system that the current position points to a characteristic point. For example, a motion pattern (e.g., pivoting), or a button press, a voice command, or simply the fact that the probe tip remains in place for a short period of time. This characteristic point can then be matched to the 3D dataset, taking into account the previous positioning of the 3D dataset relative to the reference object coordinate system. A typical characteristic point is any hard surface that can be identified in the 3D dataset, such as a planned entry point for pedicle drilling.

[0107] One method for matching characteristic points to a 3D dataset is to utilize limited-precision localization along a single DoF and a priori information about direction with limited precision. Characteristic points can be projected with limited precision in direction to find intersections with surfaces in the 3D dataset. If multiple intersections exist, and such ambiguity cannot be resolved, for example, by identifying the shortest distance or by prior information about anatomical structures, the system may prohibit the use of this point to achieve complete registration.

[0108] Another method for matching characteristic points involves knowledge of the position of the characteristic point relative to a nearby surgical instrument. For example, a scalpel placed on the skin surface may have a known distance to a characteristic point on the bone surface so that the scalpel's position can be used in the second step of registration. Similarly, placing a drill bit on the bone surface may involve a specific distance to a characteristic point or target point on the bone surface. This offset may be fixed or individually estimated based on the surgical robot's sensors or 3D dataset. Furthermore, since the system detects characteristic points in the video output / images provided by the endoscope through image processing performed by the system and estimates the position of the characteristic point relative to the endoscope, the endoscope can be a surgical instrument that can be positioned at a characteristic point, in the sense that it provides information on where the characteristic point is located in the reference coordinate system / 3D coordinate system.

[0109] In another embodiment of the two-step registration, the second step may involve acquiring a second X-ray image at a later point in time to improve accuracy. A key aspect is guidance to a position achievable by the improved registration in the second step, i.e., guidance based on incomplete registration.

[0110] For example, a second step in the registration process may be to change the position of the C-arm to acquire a second X-ray image from a different position that results in a change of at least 10 degrees in the imaging direction over the region of interest. The system can then register the second X-ray image to the first X-ray image, thus eliminating the uncertainty of incomplete registration.

[0111] In another example, the registration in the second step may require a second X-ray image from any imaging direction (i.e., no need to change the imaging direction over the region of interest in the second X-ray image), and that the surgical instrument is close to a target point within a known anatomical structure in the 3D dataset. However, guidance based on the initial registration may not be precise enough to guide to a specific target point, and will only guide to areas where characteristic points can be identified based on the second X-ray image. Once the second X-ray is acquired, the system can register the 3D dataset and reference coordinate system based on the characteristic points and the tracked surgical object visible in the second X-ray, and the tracked surgical object is tracked with respect to the reference coordinate system.

[0112] In another embodiment, based on the acquired X-ray image, the system can determine and mark target points or regions in the 3D dataset that are, for example, unsuitable for the second step of registration due to such ambiguity. The system can also determine and mark regions or points in the 3D dataset that are suitable for the second step of registration.

[0113] In another embodiment, after performing the second step of registration of the first vertebra, the system allows the use of another point on another vertebra, for example, whose relative position to the first vertebra may have changed between the acquisition of the 3D dataset and the positioning of the patient on the surgical table. The complete registration is then updated using this other point, ensuring accurate 3D navigation for the other vertebra as well.

[0114] If there is movement of anatomical structures during surgery and that movement is not being tracked by the tracking system (for example, a reference object may be firmly attached to vertebrae L1 and L2, but there is relative movement of vertebrae Th12 while inserting pedicle screws into L1 and L2, where Th12 is fully registered), it may be necessary to obtain a new X-ray image (i.e., a new first step of registration) that also depicts the current position of Th12 before performing the second step of Th12 registration.

[0115] Note that if a tracked reference object can be located within an X-ray image, it is possible to avoid reference objects attached to anatomical structures and move the same or a different object to a characteristic point that can be further tracked in its reference coordinate system and linked to a 3D dataset.

[0116] In another embodiment, the first step of registration may be of limited accuracy, resulting in the 3D dataset being registered with 3DOF relative to the reference coordinate system of the tracked object, while the remaining 3DOF can be estimated with a rotation error of up to 35 degrees or a translation error of 100 millimeters, which is therefore insufficient for precise navigation but is accurate enough for initial guidance before reaching the second step of registration. An example of a tracked object is a surgical robot arm with surgical instruments (e.g., drill bits) attached that are visible in an X-ray image. Another example may be a surgeon holding instruments that have already been registered and tracked (in a 3D coordinate system).

[0117] After the first step of registration, the system can guide the surgical object to a characteristic point. A characteristic point, such as an entry / target point for drilling a pedicle screw, is known in the 3D dataset. The system can guide the surgeon or robot toward the characteristic point. The characteristic point may be constrained by anatomical structures, for example, the drill tip stopping at a bone surface, thus constraining the area where the characteristic point can be found; or the identification of the characteristic point may be done by the surgeon understanding the characteristic point (e.g., an identifiable tip or valley of an anatomical bone structure), or by using an endoscopic camera to detect known features in the 3D dataset. The better this characteristic point is constrained, the more accurate the final 6DOF registration will be. When the surgeon or robot reaches the characteristic point and triggers an update, the system determines the 6DOF localization between the reference coordinate system and the 3D data by utilizing the correspondence between the characteristic point in the 3D dataset and the position of the surgical object in the reference coordinate system / 3D coordinate system.

[0118] In another embodiment, the surgeon can use a tracked drilling machine, and the drill bit or other attached object can be positioned in the X-ray image relative to a 3D dataset, thus having the positioning of the 3D dataset in a reference coordinate system. This positioning of the reference coordinate system relative to the 3D dataset may have a translation error of up to 50 mm in the direction of the X-ray beam and a rotation error of up to 10 degrees. Using an HMD, the system can display a 2D line, in which the surgeon is assumed to identify a characteristic point, e.g., on the skin surface. The surgeon then moves the drill tip to that position, triggering an update so that the system updates the positioning of the 3D dataset relative to the tracking reference coordinate system.

[0119] In another embodiment, the initial registration of the reference coordinate system and 3D data may include prior information. For example, 3DOF may be precisely determined in spinal surgery, while some of the remaining 3DOF may be a priori knowledge of the arrangement of anatomical structures in the reference coordinate system / 3D coordinate system. The surgery may be performed in a known patient position relative to the operating room, for example, the patient's anterior orientation may always be upward, which the system can use to estimate the position of the 3D dataset in the reference coordinate system. This allows for a rough estimate of 2DOF of rotation of the 3D dataset in the OR, which may be directly linked to the reference coordinate system, for example, in the case of a head-mounted display (HMD). Furthermore, the system may track specific body parts or operating room equipment (e.g., surgical table) to determine the approximate position or orientation of the patient, limiting the accuracy estimate of the remaining 3DOF. In addition, the surgeon can input additional information into the system, such as marking the cranial-caudal orientation of anatomical structures using an HMD, providing an estimate of at least 1DOF of the remaining 3DOF.

[0120] In another embodiment, the tracked reference object may be positioned over the region of interest before an incision is required in the surgical area. X-ray images showing the anatomical structure and the area of ​​the tracked object may be acquired, enabling 5DOF localization of the tracked object. The system guides the tracked object toward an indeterminate target trajectory, defined by a target point in a 3D dataset (e.g., bone surface beneath the skin) combined with an indeterminate direction in the reference coordinate system. It should be noted that physical contact of the surgical object with the anatomical structure may not necessarily require reaching the trajectory. A skin incision can be made near the indeterminate trajectory if necessary. The tracked surgical object can move in the direction of the indeterminate trajectory until it reaches the target point. The system can then be triggered to update a second step of registration, matching the target point between the reference coordinate system and its corresponding point in the 3D dataset to achieve 6DOF localization.

[0121] It should be noted that this allows for initial guidance on incision and further registration based on a single X-ray image. Further X-ray updates may be required if anatomical movement is anticipated.

[0122] When registering a 3D dataset to a reference coordinate system / 3D coordinate system, the reference coordinate system can be directly coupled to the reference object, or the reference object can simply be tracked within the reference system. For example, if an HMD is used to track a drilling machine, the drilling machine is typically tracked in a reference coordinate system that references the operating room. Assuming the 3D dataset is nearly static in the reference coordinate system, the guidance prior to the second registration step and the subsequent navigation can be performed in the reference coordinate system. The movement of anatomical structures can be tracked within the reference coordinate system, for example, if the tracker is directly attached to the anatomical structure registered to the 3D dataset, and as a result, the navigation of the surgical instrument can be tracked with respect to the moving anatomical structure. Another example is directly tracking a surgical table or visible anatomical structure to estimate the movement of the 3D dataset in the reference coordinate system. The embodiments described do not focus on this distinction, but they function correctly in any case where the tracked surgical object moves relative to the estimated position of the 3D dataset.

[0123] In an ideal scenario, the above-described embodiment allows for navigating, for example, pedicle drilling and pedicle screw insertion for multiple vertebrae based solely on the acquisition of a single X-ray for the entire procedure.

[0124] Example of a two-step registration workflow (Figure 1): 1. The surgeon or robot fixes a first surgical object (e.g., a vertebral tracker) to the patient's anatomical structure within the region of interest represented by a 3D dataset. 2. The first surgical object is registered and tracked in a reference coordinate system, and an X-ray image showing the first tracked surgical object and region of interest is acquired. 3. The system registers the first surgical object using the 3D dataset with limited accuracy (in other words, performs an incomplete registration). This may or may not include prior information about the 3D dataset in the reference coordinate system. 4. The second tracked surgical instrument (e.g., a drill) is tracked in the reference coordinate system. 5. The system provides guidance for a second tracked surgical instrument to a characteristic point that can be identified within the 3D dataset based on registration with limited accuracy. 6. The second tracked surgical instrument is moved towards a characteristic point and positioned at that point. Precise positioning on the point is achieved by external constraints or additional information (for example, the bone surface may be detected by the surgeon or robot by pressure, visual cues, or knowledge of anatomical structure). 7. The system updates the registration between the reference coordinate system and the 3D dataset based on the location of a second tracked surgical instrument that matches a characteristic point in the 3D dataset.

[0125] Example of a two-step registration workflow (Figure 1): 1. The surgeon or robot holds a first surgical instrument (e.g., a drilling machine with a drill bit attached) on a region of interest represented in a 3D dataset (e.g., on the skin and on the vertebrae of the spinal region). 2. The first surgical instrument is tracked in a reference coordinate system, and an X-ray image showing the tracked first surgical instrument and region of interest is acquired. 3. The system registers the 3D dataset to the reference coordinate system with limited accuracy. 4. The system provides guidance regarding a second tracked surgical instrument (which may or may not be the first surgical instrument) near a characteristic point on the bone surface that can be identified in the 3D dataset. The guidance provides directional information, which is limited in accuracy to the system. The system can indicate a line of a reference coordinate system, on which the system predicts the location of the characteristic point, which can be visualized, for example, using an HMD. 5. The second tracked surgical instrument is moved near the area of ​​limited precision. The surgeon or robot sets an incision on the skin, and a characteristic point on the bone surface beneath it is indicated by the system's guidance. 6. Surgical instruments are moved through the incision onto the bone surface. Precise placement at a point is achieved by external constraints or additional information (for example, the bone surface may be detected by the surgeon or robot by pressure, visual cues, or knowledge of anatomical structure). 7. The system updates the registration between the reference coordinate system and the 3D dataset (providing complete registration) based on the position of a second tracked surgical instrument that matches characteristic points on the bone surface in the 3D dataset.

[0126] System example Figure 2 shows an embodiment of a system including a C-arm-based X-ray imaging device 200, a movable patient table 300, and a robotic device 400. Also shown in Figure 2 is a physician 100 at the control console. Physician 100 can operate control input devices 110 and 120, which may be any type of input device such as a joystick or a three-dimensional motion receiver. Physician 100's hand movements may be transmitted to the robotic device 400, particularly to the part of the robotic device closer to the patient on the patient table 310. Meanwhile, by moving their hand on input devices 110 and 120, physician 100 can control the position and orientation of other devices such as the X-ray imaging device 200 and the patient table 300. It will be understood that physician 100 may be in a remote room or directly in the operating room. Furthermore, physicians may utilize other input devices such as voice control, or a keyboard or mouse to control various devices in the operating room.

[0127] The C-arm-based X-ray imaging device 200 comprises an X-ray source 210 and an X-ray detector 220. In the illustrated embodiment, the X-ray imaging device is mounted by a series of movable arms 230 so that the positions of the X-ray source and detector can move relative to the patient. Similarly, the patient table 310 is mounted by a series of movable arms 320 so that the position of the patient table can also be changed. The robotic device 400 is shown with several robotic arms 410, including a joint 420 that allows free movement of the end effector 430.

[0128] This disclosure encompasses a system including cameras and tracking devices, although these devices are not shown in Figure 2. However, the physician 100 in Figure 2 is wearing an augmented reality device in the form of a pair of glasses 130. When the physician is near the patient table 310, the augmented reality device 130 can provide tracking capabilities. Furthermore, such a device 130 can provide information and visualizations on the glasses, which function as a display. As an alternative possibility to the display for providing information and / or visualizations, a monitor 140 is schematically shown in Figure 2.

[0129] Examples of supported 3D navigation Figure 3 is a 2D X-ray projection image of the spine from the anterior-posterior direction. The surgical object (1.SO) is positioned so that its tip lies on the bone surface of vertebra T8 (1.T8). Two contours of the surgical object are shown with respect to two different 3D positions (1.P1 and 1.P2, respectively) and 3D orientations of the surgical object, and the contours of the surgical object (white solid lines labeled 1.O1 and 1.O2, respectively) are nearly identical. The tip of the surgical object in 3D space lies on the line defined by the drill tip and the X-ray source of the X-ray machine (typically the focal point), i.e., on the epipolar line of the tip of the surgical object. Depending on the 3D position of the tip of the surgical object, the 3D orientation can be adjusted so that the projected contour of the surgical object fits its appearance in the 2D X-ray image. It is emphasized that there are countless combinations of 3D orientation and corresponding 3D positions that result in nearly identical contours of the surgical object, two of which are depicted in Figure 3.

[0130] Figure 4 shows a slice of the same 3D scan of the spine as in Figure 3, depicting a sagittal section. The white line (2.EL) indicates all possible 3D positions of the tip of the surgical object, corresponding to the epipolar line of the tip of the surgical object from Figure 3. Two points on the line were selected, one (2.P1) where the tip of the surgical object is located on the bone surface of vertebra T8 (2.T8), and the other (2.P2) where it is at a specific distance from the bone surface. It is clearly visible that the 3D orientation of the surgical object differs depending on the 3D position of the surgical object, but both have nearly identical contours in the X-ray projection image in Figure 3.

[0131] Figure 5 shows a slice from the same 3D scan as Figure 4, depicting an axial cross-section. The white line (3.EL) indicates all possible 3D positions of the tip of the surgical object, corresponding to line 2.EL in Figure 4. As in Figure 4, two points on the line are selected: one point (3.P1) where the tip of the surgical object is located on the bone surface of vertebra T8 (3.T8), and the other point (3.P2) where it is at a specific distance from the bone surface. Here again, although the 3D orientation of the surgical object is different, it can be seen that both have nearly identical contours in the X-ray projection image in Figure 3.

[0132] In other words, Figures 3–5 show two different 3D constellations with two different drill poses, which produce identical (or nearly identical) X-ray projection images. The two drill constellations differ in their imaging depth (distance from the image plane, i.e., from the X-ray receiver) and inclination (i.e., 3D orientation). However, if it is possible to eliminate ambiguity regarding imaging depth (by some prior information), the drill pose relative to the patient's anatomical structure can be uniquely determined.

[0133] This disclosure teaches how to eliminate ambiguity regarding imaging depth by establishing a correspondence between the anchor points of a surgical object and the target points (whose positions in a 3D coordinate system defined by a 3D dataset are known) by utilizing prior information that the target points and anchor points are on the bone surface or at a defined distance from the bone surface. This can be done based on a single X-ray image.

[0134] Alternatively, ambiguity regarding imaging depth can be eliminated without using target points. According to one embodiment, the tip of the drill (i.e., the anchor point) is positioned on the bone surface near the intended drill path (i.e., within the region of interest), and an X-ray image is acquired. The position of the drill tip is detected in the X-ray image. For the imaging direction on the region of interest, determined based on DRR matching of the 3D dataset to the X-ray image, a line (so-called epipolar line) in the 3D coordinate system defined by the 3D dataset is determined by the possible drill tip positions in 3D space on the detected anchor point, which are virtual projections onto the X-ray image. Since there is prior knowledge that the drill tip is on the bone surface, the position of the drill tip in the coordinate system defined by the 3D dataset can be found by determining the point on the line that intersects (i.e., where it is located) the bone surface. Finding the bone surface along the line may be done automatically, for example, by evaluating the Hounsfield values ​​in the 3D dataset along (or in the vicinity of) the line and searching for strong gradients. Of course, if local segmentation of the bone surface within the relevant region is already available, this may be used to intersect the line.

[0135] By acquiring additional X-ray images from the current imaging direction and / or different imaging directions, it may be possible to improve the accuracy of determining the spatial relationship between the surgical object and the region of interest.

[0136] It is also possible to determine the 3D spatial relationship of the surgical object to the region of interest after it has been inserted into the patient, i.e., after the anchor point is no longer located on the bone surface. This is important, for example, when performing drilling. In such a case, an X-ray image is acquired at a first time point, from which the drill start point in the 3D data coordinate system is determined, and it is assumed that this is the 3D position of the drill tip (i.e., the anchor point) at the first time point. After the instrument has been advanced into the patient, a further X-ray image is acquired at a second time point. Here, ambiguity regarding the 3D drill pose can be resolved based on the assumption that the drill axis determined at the second time point passes through the drill start point determined at the first time point. This assumption requires that drilling actually begins at the anchor point determined at the first time point. This assumption is particularly easily justified when a surgical robot is used, as it eliminates unintended movement of the surgical object and allows patient movement to be detected by single-image registration. Patient movement can also be detected in real time by a head-mounted augmented reality device (for example, using a camera integrated into the augmented reality device). At any given time, if the system detects patient movement, it can request new X-ray images, request a repetition of the procedure, or use the detected movement of the anatomical structure to improve the estimate of the drill pose relative to the anatomical structure.

[0137] If the patient's movement is sufficiently large compared to the expected movement of the drill bit, simply detecting the movement of the drill bit (without tracking an independent object) may be sufficient to detect that the patient has moved. This can be used to alert the surgeon, for example, by estimating that the drill may have slipped from its desired position.

[0138] Similarly, if the respiratory motion is sufficiently large compared to the drill tip motion, it may even be sufficient to identify the respiratory motion as a component of the drill tip motion and to model the respiratory motion as a periodic motion that can be subtracted from the drill tip position to improve the accuracy of the 3D position of the drill tip relative to the moving anatomical structure.

[0139] An alternative, potentially more accurate, method for tracking patient movement is to track a second object to improve the accuracy of drill tracking relative to anatomical structures within the region of interest. For example, the system could directly track the patient (e.g., visible skin or tissue around the entry point of the drill bit), while an indirect example would be estimating the position of a sleeve independently of the drill bit, which could be assumed to remain on the bone surface during drilling, allowing for the detection of drill depth relative to the moving bone surface. Overall, directly or indirectly tracking the anatomical region of interest can reduce noise in tracking drill position relative to the region of interest and improve the accuracy of surgical steps (e.g., with respect to drill guidance for surgeons or surgical robots).

[0140] Tracking of objects such as drills, drilling machines, sleeves, or patient anatomical structures may be performed in real time or at any given point in time. For example, tracking may not be available throughout the entire time due to obstruction, placement of the object resulting in unacceptable ambiguity, or spontaneous interruption of the tracking process.

[0141] To utilize drill tracking, it is beneficial to link the drill to a 3D coordinate system of 3D data or a 3D coordinate system of an X-ray image. For this purpose, it is necessary to track the object visible in the X-ray outside the X-ray to estimate the spatial relationship between the tracking system and the other 3D coordinate system. This may require the systems to be somewhat synchronized so that the tracked drill can be mapped to the detected drill in a specific X-ray. Alternatively, to generate a sufficiently accurate spatial relationship, it may suffice to estimate the patient's orientation and position based on prior information (including a calibration step). Coordinate system registration may be approached in different ways than the center of the system disclosed herein.

[0142] This procedure may be repeated, and in some cases, X-ray images may be acquired from other imaging directions, for example, to improve accuracy.

[0143] When the system directly controls the imaging device, it can automatically acquire new images. When the system controls the robotic imaging device, it can automatically acquire images from the desired imaging direction.

[0144] According to one embodiment, such a system may be particularly useful in spinal surgery. The following three workflows are examples of how this system can be used to perform drilling for pedicle screws for spinal fixation.

[0145] An example workflow for drilling pedicle screws according to the plan (see Figure 6). Note that the step numbering in the workflow in Figure 6 is a combination of "1." indicating that this is the first of three workflows, followed by the actual step number used in the following description of the workflow. 1. The surgeon and / or system perform preoperative planning in a 3D dataset (e.g., CT scan or 3D X-ray scan) by determining the target path, i.e., the planned drill trajectory, in the 3D dataset, where the target point is the starting point of the intended drill trajectory located on the bone surface of the pedicle, and the target end point is the end point of the intended drill trajectory. 2. The surgeon positions the tip of the drill on the dorsal bone surface approximately near the pedicle of the vertebra to be drilled (the drill tip can be positioned more precisely if the relative position of the drill to the bone is already known). 3. If the system utilizes external real-time (or near real-time) tracking, the system can continuously track the drill bit, drilling machine, or sleeve to determine the drill position at least partially independently of X-rays. If a tracking system is unavailable, you may proceed to step 5. 4. The system can continuously detect a second reference (e.g., tracking an anatomical structure or sleeve within the region of interest) and / or model the movement of the region of interest to improve the tracking accuracy of the drill bit relative to the region of interest. 5. The surgeon acquires X-ray images (for example, in an approximately anterior-posterior (AP) imaging direction), preferably holding the drill so that the surgeon's hand does not enter the X-ray beam. If the imaging direction causes the surgeon's hand to enter the X-ray beam, the system can provide instructions for a different imaging direction. 6. The system detects the location of the drill tip (i.e., anchor point) in the X-ray image. The system localizes the region of interest in the X-ray image by calculating a digitally reconstructed radiograph (DRR) from the 3D dataset for a number of possible localizations. The best-matching localization of the DRR is selected. The DRR does not need to be limited to the region of interest. The system can use a weighting function to more accurately match important areas within the 2D region of interest (e.g., if the image quality is poor), and the 2D region of interest is determined by the X-ray drill tip position. The weighting function may also depend on the C-arm type (flat panel or image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D dataset (preoperative or intraoperative). Based on the localization, the system determines a virtual projection of the target point into the X-ray image, which may be displayed as an overlay in the X-ray image. 7. The system determines the 3D drill tip position in the coordinate system of the 3D dataset based on the detected drill tip in the X-ray image by applying the knowledge that both the drill tip and the target point are on the bone surface. If the distance between the drill tip (i.e., the anchor point) and the target point is below a threshold (e.g., 2 mm, the threshold may depend on the type of vertebra), the system proceeds to step 8. Otherwise, the system instructs the surgeon to reposition the drill bit to move it closer to the target point. The surgeon follows the instructions and returns to step 5. 8. When the drill tip is still some distance from the target point, the local model of the vertebral surface near the target point can be taken into consideration when determining the 3D drill tip position. The local model may be derived from a vertebral model if the vertebra to be drilled has already been classified (i.e., which vertebral level it is, e.g., L3, L4, etc.). 9. Based on the determined 3D drill tip position in the coordinate system of the 3D dataset, the system determines the 3D position and orientation of the drill in the coordinate system of the 3D dataset. 10. If the deviation between the 3D drill orientation and the target path is below a threshold (e.g., less than 2°), proceed to step 12. Otherwise, the system issues an instruction to adjust the drill angle. 11. The surgeon adjusts the drill angle and returns to step 5 (without changing the imaging direction), or continues / starts drilling directly and continues drilling in step 13. While the surgeon adjusts the drill angle, the system can update information (e.g., trajectory, position, angle, etc.) in real time or near real time. 12. The system calculates the remaining drill depth and issues the corresponding drill instruction. 13. The surgeon follows instructions (e.g., by drilling a predetermined distance). During the drilling procedure, the system provides information on the current drill depth in real time or near real time. Whenever the system desires or recommends checking the drill trajectory (e.g., after drilling a certain distance or when drilling near an important anatomical structure), a new X-ray image can be acquired (without changing the imaging direction). 14. The system performs matching, detection, etc., as described above. Based on the actual drill start point (for example, the position of the drill tip shown in the X-ray image when the first drill command is issued), the system determines the 3D position and 3D orientation of the drill. 15. If the remaining drilling distance is below a threshold (e.g., 1 mm), proceed to step 17. If the deviation between the orientation of the 3D drill and the target path is below a threshold (e.g., 1°), return to step 12. Otherwise, the system will instruct the system to adjust the drilling angle. If there is drilling that is too deep, the system may issue a warning. 16. The surgeon follows instructions, obtains new X-ray images, and returns to step 14. 17. The current drilling procedure for the pedicle is complete. 18. Return to step 2 to perform the next pedicle drilling.

[0146] It should be noted again that all references to "surgeon" in this workflow may refer to a human user, a robotic surgeon, a mechanical support device, or a similar device.

[0147] In general, in addition to each of the exemplary workflows disclosed herein, the system can also provide support for performing skin incisions. This may be particularly useful when a surgical robot or robotic arm is used. According to one embodiment, a surgical instrument (e.g., a scalpel) is aligned along a target trajectory, but at a sufficiently large distance from the target point so that the instrument does not yet touch the skin. The robot or robotic arm can then move the surgical instrument along the target trajectory toward the target point using a soft tissue protection sleeve until it encounters some resistance (i.e., it touches the skin). The resistance may be automatically detected, for example, by a pressure sensor. The incision may be made through the soft tissue protection sleeve. After the incision has been made, the surgical instrument can be changed (e.g., switched to a drill while holding the soft tissue protection sleeve in place), and the surgical instrument can be advanced until it encounters greater resistance (i.e., it touches the bone surface). The resistance may be automatically detected, for example, by a pressure sensor. After completing drilling on one pedicle, the procedure can be repeated for the next pedicle. The pressure sensor may be incorporated into the power tool or robotic arm that holds the surgical object. By recording the pressure exerted by the surgical object on the anatomical structure, the system can distinguish, for example, between a drilling process and a slipping motion in which the surgical object slides over the anatomical structure.

[0148] It may be useful if a robot or robotic arm could automatically switch surgical instruments (possibly within a soft tissue protection sleeve), advance or retract the soft tissue protection sleeve, and drill through the soft tissue protection sleeve (possibly using a vibratory drill mode). It may be advantageous to specially design a soft tissue protection sleeve or adapter suitable for the robot. The design of such a sleeve may also take into account detectability in radiographic images. For example, it may be advantageous to fabricate the sleeve from a radiolucent material, or partially from a radiolucent material (e.g., in the area covering the tip of the drill), so that the surgical object (e.g., a drill, especially its tip) is visible in radiographic images even when covered by the sleeve. It may also be advantageous to incorporate certain features into the surgical object and / or sleeve that can be used to make the detection of the surgical object in radiographic images easier.

[0149] The sleeve can also be patient-specific if partial segmentation around the target point is performed preoperatively. The sleeve may be constructed, for example, by a 3D printer and may or may not be made from a transpene material. The tip of the patient-specific sleeve (the geometric shape of the opening may be a closed line in 3D space) can fit to the surface of the patient's bone anatomical structure around the target point, based on the precise rotational alignment of the sleeve. This may have the advantage of a very good fit and may further eliminate the need for precise navigation information, as it helps the surgeon (or robot) to find the desired position of the sleeve tip on the planned target point with great precision. This can also ensure that the sleeve tip is not positioned in an incorrect 3D position, which can occur when there is surface ambiguity based on suboptimal imaging orientation, meaning that there are two points on the surface with different 3D coordinates, but appear at the same 2D coordinates in the X-ray image.

[0150] Similar techniques may also be applied to position the endoscope (the trocar of the endoscope may have a specific extension at its tip) at a desired distance to the surface of the anatomical structure of the patient's bones, and thus, although it is possible to use X-ray images from a single line of sight, it may still be possible to precisely position the endoscope at a desired 3D position above the surface of the bone.

[0151] Another technique to address the aforementioned potential ambiguity problem may be to determine the direction of the X-ray beam through the planned 2D position or current 2D position of the target point, based on the X-ray image and the determined line of sight direction. After determining the 5DOF position of the region of interest in the X-ray image (i.e., after determining the transformation between the X-ray image 3D coordinate system and the dataset 3D coordinate system), this beam, which is a line in the dataset's 3D coordinate system, intersects the 3D dataset. In the case of a 3D dataset that can provide density information (e.g., CT), all points (voxels) that intersect this line indicate a density value (e.g., a Hounsfield scale). By analyzing all the density values ​​of the intersecting voxels of this line (e.g., analyzing the gradient), it can be determined whether there are two or more intersections of the line with the bone surface in the neighborhood (e.g., less than 10 mm) around the target point. In this case, a different imaging direction may be required to avoid ambiguity and compute accurate 3D navigation information. This procedure may be repeated for multiple imaging directions to find an imaging direction suitable for a particular target point. During the procedure, the system can advise the surgeon or robotic surgeon to properly position the imaging device to prevent inaccurate navigation based on the above-mentioned topological ambiguity.

[0152] The sleeve may be semi-transparent in X-rays around its tip, but not semi-transparent further away from the tip (for example, 40 mm above the tip). In this case, the system can detect the drill bit inside the sleeve, while still being able to determine the 3D position of the sleeve relative to the region of interest.

[0153] A fully X-ray semi-transparent sleeve can be tracked, for example, by a real-time tracking system, and based on the determination of the 3D position of the drill bit based on an X-ray image at a first time point and the knowledge that the drill was inserted into the sleeve at that first time point. The 3D position of the sleeve relative to the region of interest can be determined at a second time point. The system may be able to do the same even if the sleeve is partially X-ray semi-transparent, and the opaque portion of the sleeve that is not sufficiently opaque is present in the X-ray image to determine its 3D position relative to the region of interest.

[0154] According to one embodiment, the use of a surgical robot may also make it possible to address the movement between individual vertebrae caused by the patient's respiration. This respiration can be modeled and detected by a pressure sensor. The robot can maintain constant pressure on the vertebrae when not drilling.

[0155] This procedure may also be applied, for example, during wedge osteotomy to determine the position of the chisel, in which case the midpoint of the chisel is used as the anchor point.

[0156] A similar procedure is applicable when using soft tissue protection sleeves that are not made of radiopaque material, where anchor points (e.g., the tip of a drill) may not be visible in the X-ray image. In such cases, a virtual anchor point can be used instead of the (invisible) anchor point. This virtual anchor point can be determined based on a previously acquired X-ray image in which the anchor point was visible, and on the assumption that there was no movement of the anchor point between the generation of the current X-ray image and the generation of the previously acquired X-ray image. Based on the current X-ray image, the plane is defined by the axis of the sleeve and the center of the X-ray beam. The virtual anchor point can be determined by orthogonally projecting the anchor point from the previous X-ray image onto that plane. The virtual anchor point can also be determined by selecting the nearest point from the anchor point from the previous X-ray image to a contour in the plane (e.g., the intersection of the plane and the bone model, or partial 3D segmentation of the bone surface) to ensure that the virtual anchor point is located on the bone surface. Even if an anchor point or anchor morphology is not visible in any X-ray, it may still be detectable, for example, if other prominent points provide sufficient information. For example, a drill bit may have a identifiable groove that is sufficient to project the location of a virtual anchor point.

[0157] When a sleeve is used, the sleeve itself can be considered a second surgical object that helps determine the 3D spatial relationship of the drill bit (surgical object) to the region of interest. The sleeve can be detected in the X-ray image, and the fact that the drill bit is inserted into the drill sleeve (i.e., the axis of the drill sleeve constrains the axis of the drill bit) can be utilized. It may also be possible to track the drill sleeve using augmented reality devices (such as head-mounted augmented reality glasses).

[0158] In the following, we will consider this an example of "aligned" surgical instruments, where the drill and sleeve shaft are constrained to each other, even though it may still be necessary to consider the possibility of bending or oscillating within the sleeve.

[0159] Another application of the procedure may be tumor resection. In this case, a preoperatively acquired MRI scan (e.g., including the tumor contour or planned resection volume) may be fused with a preoperatively or intraoperatively acquired 3D X-ray scan, and one or more target points are identified in the MRI scan and / or 3D X-ray scan. The surgical instrument may be a resection instrument with anchor points. By determining the relative 3D position and orientation of the resection instrument and the tumor contour (or planned resection volume), accurate 3D navigation instructions can be provided even if several target points are defined at planning time. The resection may also be performed at a fixed distance from the target points. The use of a robot can also improve accuracy. The robot can receive relative 3D positioning information from, for example, an internal sensory device, so that the 3D position and orientation of the instrument relative to the resection volume are available at any point after the final determination based on the acquisition of X-ray images. Verification of this information is possible at any point in time by acquiring additional X-ray images, even without using anchor or target points. A virtual X-ray image is generated, assuming the imaging direction has not changed, based on the available information from the positioning sensory device and the final determination of the instrument's 3D position and 3D orientation relative to the excised volume based on the last X-ray image. After acquiring additional X-ray images from the same imaging direction, the additional images are compared to the virtual X-ray image, and the algorithm determines whether the similarity is sufficiently high (positioning information is still valid) or (there may have been unknown movement). In the latter case, a new image with determined anchor and target points may be required.

[0160] This procedure may also be combined with a real-time navigation and tracking system. Since this procedure is the same as existing techniques and offers a high level of diversification, such a combination would result in extremely rigorous navigation, and this level of safety could enable autonomous robotic surgery.

[0161] An exemplary workflow for drilling pedicle screws (see Figure 7) (without planning the target point, single image, but with knowledge of the bone surface). Similar to the previous workflow, the step numbering in the workflow in Figure 7 is a combination of "2." indicating that the exemplary workflow is the second of three, and the actual step numbers used in the following description of the workflow. 1. The surgeon positions the tip of the drill on the dorsal bone surface approximately near the pedicle of the vertebra to be drilled. 2. The surgeon acquires an X-ray image (e.g., approximately anterior-posterior (AP) imaging direction) and preferably holds the drill so that the surgeon's hand does not enter the X-ray beam. If the imaging direction causes the surgeon's hand to enter the X-ray beam, the system can provide instructions for a different imaging direction. 3. The system detects the position of the drill tip (i.e., anchor point) in the X-ray image. The system localizes the region of interest in the X-ray image by calculating digitally reconstructed radiographs (DRRs) for multiple imaging directions from the 3D dataset (this may have already been done before acquiring the X-ray image). To localize the region of interest, the DRR that best matches the X-ray image is selected. The DRR does not need to be limited to the region of interest. The system can use a weighting function to more accurately match important areas within the 2D region of interest (e.g., if the image quality is poor), and the 2D region of interest is determined by the position of the X-ray drill tip. The weighting function may also depend on the C-arm type (flat panel or image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D dataset (preoperative or intraoperative). The system determines the lines in the 3D dataset defined by the anchor point and the direction of the X-ray beam. The system determines the point on the line where it intersects the bone surface (for example, by starting from the last point of the line, observing the voxels along this line, and determining the voxel with the highest grayscale gradient on this line, and possibly around this point). This point defines the location of the anchor point in the 3D dataset. 4. Based on the determined 3D drill tip position in the coordinate system of the 3D dataset, the system determines the 3D position of the drill in the coordinate system of the 3D dataset. Based on the 3D dataset, the system can display the current position in different views, such as axial and sagittal views, and extend the drill trajectory to better visualize the corresponding drill path. The system can also display detected anatomical structures, such as bone cortex, and indicate measurements of those structures or suggest boundaries, such as cortex near buffer zones that should not be crossed. 5. The surgeon can adjust the position and / or angle of the drill tip and return to step 2 (without changing the imaging direction), or proceed to step 6 by directly starting / continuing drilling. While the surgeon adjusts the position and / or orientation, the system updates the current position and orientation in different views (e.g., axial and sagittal) in real time or near real time. 6. To improve accuracy, new X-ray images may be acquired whenever it is desired to verify the drill trajectory, or whenever recommended by the system (for example, after drilling a certain distance, or when drilling near an important anatomical structure) (no change in imaging direction is necessary). 7. The system performs matching, detection, etc., as described above. Based on the actual drill start point (e.g., the position of the drill tip shown in the X-ray image when the first drill command is given), the system determines the 3D position of the drill. Based on the 3D dataset, the system displays the current position of the drill in different views, such as axial view and sagittal view. If the surgeon does not consider the current position to be good, the surgeon may adjust the angle (e.g., while the drill bit is in motion) and proceed to step 6. 8. The system provides information on the current drill depth in real time or near real time. Based on this information, the surgeon determines whether the current pedicle drilling procedure is complete. 9. Return to step 1 to perform the next pedicle drilling.

[0162] Here again, all references to "surgeon" in this workflow may refer to a human user, as well as a robotic surgeon, mechanical support device, or similar equipment.

[0163] Exemplary workflow for drilling pedicle screws (without target point planning, dual image) (see Figure 8) Similar to the previous workflow, the step numbering in the workflow in Figure 8 is a combination of "3." indicating that the exemplary workflow is the third of three, and the actual step numbers used in the following description of the workflow. 1. If the system utilizes external real-time (or near real-time) tracking, the object can be tracked continuously when available. Possible objects include drill bits, drilling machines, or sleeves for determining the drill position at least partially, independently of X-rays. If a tracking system is not available, you may proceed to step 3. 2. Furthermore, since the sleeve tip can always remain on the bone surface regardless of respiratory movement, the system continuously detects a second reference (e.g., an anatomical structure within the region of interest, or tracking of the sleeve positioned on the bone surface of an anatomical structure within the region of interest) and / or models the movement of the region of interest to improve the accuracy of tracking the drill bit relative to the region of interest, so that, for example, the patient's respiratory movement can be distinguished from the drill's penetration depth. 3. The surgeon positions the tip of the drill on the dorsal bone surface approximately near the pedicle to be drilled. The surgeon then begins drilling a short distance (e.g., 3 mm) in approximately the desired direction. 4. The surgeon acquires an X-ray image (e.g., approximately anterior-posterior (AP) imaging direction) and preferably holds the drill so that the surgeon's hand does not enter the X-ray beam. If the imaging direction causes the surgeon's hand to enter the X-ray beam, the system can provide instructions for a different imaging direction. 5. The system detects the position of the drill tip (i.e., anchor point) in the X-ray image. The system localizes the region of interest in the X-ray image by calculating digitally reconstructed radiographs (DRRs) for multiple imaging directions from the 3D dataset (this may have already been done before acquiring the X-ray image). To localize the region of interest, the DRR that best matches the X-ray image is selected. The DRR does not need to be limited to the region of interest. The system can use a weighting function to more accurately match important areas within the 2D region of interest (e.g., if the image quality is poor), and the 2D region of interest is determined by the position of the X-ray drill tip. The weighting function may also depend on the C-arm type (flat panel or image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D dataset (preoperative or intraoperative). 6. The surgeon takes further X-ray images. The angle of the drill may change between the images taken in step 4 and these images, but it is necessary to ensure that the drill tip stays in place, which may be easy as the drill tip is already inside the bone (see step 3). Alternatively, the surgeon may be drilling between the first X-ray image and the further X-ray image, while keeping the drill axis of the further X-ray near the anchor point (where the tip was positioned) of the first X-ray. 7. The system detects the location of the drill tip (i.e., the anchor point) in the subsequent X-ray images and localizes the region of interest in the subsequent X-ray images. 8. The system registers both X-ray images based on the localization of the region of interest. Based on this image registration and the multiple detected drill tips, the system calculates the epipolar lines of the detected drill tips and determines the 3D drill tip position in the coordinate system of the 3D dataset by calculating the nearest point between the epipolar lines. If the surgeon drills between the first image and the subsequent image, the epipolar lines will be further apart. However, the axis of the drill in the subsequent image represents a plane in the 3D coordinate system, and its intersection with the epipolar line of the drill tip in the first X-ray provides the 3D drill tip position in the first X-ray image. This is sufficiently accurate if the line of sight to view the drill changes sufficiently, for example, between 10 and 120 degrees between the first image and the subsequent image. Proceed to step 9 if successful, otherwise proceed to step 6. 9. Based on the 3D drill tip position determined (from at least the first image) within the coordinate system of the 3D dataset, the system determines the 3D position and orientation of the drill within the coordinate system of the 3D dataset (from both images). Based on the 3D dataset, the system can display the current position in different views, such as axial and sagittal views, and extend the drill trajectory to better visualize the corresponding drill path. 10. The surgeon can adjust the position and / or angle of the drill tip and return to step 4 (without changing the imaging direction), or directly start / continue drilling and proceed to step 11. While the surgeon adjusts the position and / or orientation, the system updates the current position and orientation in different views (e.g., axial and sagittal) in real time or near real time. 11. To improve accuracy, new X-ray images may be acquired whenever it is desired to verify the drill trajectory, or whenever recommended by the system (for example, when using a robot for drilling) (e.g., after drilling a certain distance, or when drilling near an important anatomical structure) (no change in imaging direction is necessary). 12. The system performs matching, detection, etc., as described above. Based on the actual drill start point (e.g., the position of the drill tip shown in the X-ray image when the first drill command is given), the system determines the 3D position and 3D orientation of the drill. Based on the 3D dataset, the system displays the current position of the drill in different views, such as axial view and sagittal view. If the surgeon does not consider the current position to be good, the surgeon can correct the angle (e.g., while the drill bit is in motion) and proceed to step 11. 13. The system provides information on the current drill depth in real time or near real time. Based on this information, the surgeon determines whether the current pedicle drilling procedure is complete. 14. Return to step 3 to perform the next pedicle drilling.

[0164] Matching of 2D-2D virtual X-ray images and actual X-ray images When generating multiple virtual X-ray images from a 3D dataset, six degrees of freedom can be considered: three for translation and three for rotation. These parameters may or may not be correlated. To reduce the number of generated images and improve runtime, it may be desirable to distinguish between two 3D rotation parameters and four 3D image parameters. The 3D image parameters do not affect the actual appearance of anatomical structures in the virtual X-ray image, but only their position; that is, the four 3D image parameters change the x and y scale, rotation, and translation (in image coordinates). The remaining two 3D rotation parameters change the appearance of anatomical structures in the virtually generated X-ray image. For this reason, it may be advantageous to generate virtual X-ray images from a 3D dataset using only two 3D rotation parameters (e.g., in 2-degree increments from -30 degrees to +30 degrees, depending on the range to be covered, the accuracy to be achieved, or the runtime to be achieved).

[0165] The system can generate multiple virtual X-ray images according to the above description and compare all virtual images with actual X-ray images acquired by an X-ray imaging device. The system can find specific feature points (e.g., edges, corners, etc.) in the X-ray images and similarly in all virtual images. By comparing the features of the actual X-ray images with the features of each virtual image, the system can find the optimal transformation so that the virtual images fit the actual X-ray images as well as possible. If the viewing direction differs between the virtual images and the actual X-ray images, or if the actual X-ray images depict different parts of anatomical structures than the virtual images, the virtual images will not fit the actual X-ray images even after applying the image transformations discovered by the feature detection algorithm. This can be detected by the system using an appropriate metric, such as an image similarity metric (for example, gradient vector correlation can be an appropriate metric). If, among the multiple virtually generated X-ray images, there is one image that shows at least some similar parts of anatomical structures from a viewing direction that is somewhat similar to the actual X-ray image, the feature detection algorithm provides an image transformation matrix that describes how the virtual images should be transformed to best fit the actual X-ray images. This transformation may also be described by the four 3D image parameters mentioned above. Knowing the remaining two 3D rotation parameters from which the virtual image was generated, a complete set of parameters is obtained to determine all six degrees of freedom for registering the 3D dataset to an actual 2D X-ray image.

[0166] The number of parameters used to generate multiple virtual X-ray images and the number of parameters used for feature detection-based transformations can be changed. For example, it may be possible to use four parameters for generating multiple virtual X-ray images (e.g., two 3D rotation parameters, image rotation, and image zoom) and the remaining two parameters for the feature detection algorithm (e.g., translation within the image plane).

[0167] It should be further noted that the total number of parameters used may exceed the number of degrees of freedom. For example, the system uses image rotation (or any other parameter) in its feature detection algorithm and when generating multiple images, thus potentially resulting in a total of seven parameters (or more if more parameters are used twice).

[0168] To date, feature detection algorithms that detect and evaluate features on actual X-ray images are independent of feature detection on multiple virtual X-ray images. Alternatively, algorithms may detect features on actual X-ray images and find corresponding features on multiple virtually generated images, leading to dependent feature detection between actual and virtual images. Similarly, it may be possible to detect specific features in one of the virtual images and find these features in the actual X-ray projection image.

[0169] It should be noted that the system does not necessarily compare all image features from multiple virtually generated images with those of the actual X-ray image. It may be sufficient for the system to simply stop the comparison when it finds a corresponding image, thus skipping the remaining images from the multiple virtual images.

[0170] Depending on the step size used during the generation of multiple virtual X-ray images, this result may still be insufficient in terms of accuracy. Therefore, the system can generate additional virtual images, for example, by starting from the current result and using a smaller step size and / or a smaller parameter range. Then, by repeating the procedure (including feature extraction, feature comparison, image transformation, etc.) with these new virtual X-ray images, more accurate results can be obtained.

[0171] Alternatively or additionally, optimization algorithms may be used to find more accurate results. An optimizer can generate a virtual X-ray projection image given a set of parameters describing six degrees of freedom. The optimizer can use a loss function that translates a given set of parameters into a loss (e.g., one or more values). For example, this loss could be a metric describing the similarity between two images (e.g., mean squared error, structural similarity, gradient-vector correlation). For example, the loss can be calculated by transforming a 3D dataset based on a given set of parameters, generating a virtual image based on the transformation, and comparing the virtual image to the actual X-ray image. If the loss of the optimization algorithm (e.g., an image similarity metric) is not mathematically derivable, the algorithm can change one of the six parameters, generate further virtual X-ray images, and repeat this step for all six parameters. Thus, the optimization algorithm can choose which direction to individually change the parameters in to achieve a smaller loss. This process is iterative and may be stopped when the loss and / or parameters no longer change significantly.

[0172] Alternatively, if the loss can be mathematically derived, it can be used to determine the Jacobian matrix (or higher-order derivative matrix), which helps determine how the optimizer should change its parameters in the next iteration to achieve a smaller loss.

[0173] Note that optimization algorithms may also use fewer than six degrees of freedom if one or more degrees of freedom are fixed by prior information or other knowledge. Furthermore, note that both X-ray images and / or multiple X-ray images can be transformed into a different space (e.g., by applying a 2D Fourier transform) before feature extraction, so that feature extraction and feature detection may provide improved results. Subsequent steps (i.e., the step of determining the image transformation matrix and six parameters, etc.) may remain the same. Alternatively, the 3D dataset may be transformed into a different space before generating multiple virtual X-ray images (e.g., to highlight other features that can be found more easily).

[0174] Exemplary workflow for registration with a 3D dataset of anatomical structures (Figure 9): 1. The system receives a 3D dataset of anatomical structures, generates multiple virtual X-ray images considering two 3D rotation parameters that do not describe image rotation, and detects features in all images. 2. The system receives a 2D X-ray projection image and detects features within it. 3. The features of the X-ray images are compared with the features of each virtually generated image, and the optimal 2D transformation is determined for each image pair. 4. The system evaluates the similarity between the actual X-ray image and all transformed virtual images according to the determined 2D transformation. The system selects the virtual image that best fits the real image. All six degrees of freedom are determined considering the two 3D rotation parameters used to generate the virtual image and the 2D image transformation. 5. Optional: To improve accuracy, the system can generate multiple virtual X-ray images starting from the results of step 4 and repeat steps 3 and 4 for the new multiple images. 6. Optional: The system may use an optimization algorithm that can optimize all degrees of freedom (or fewer, if one or more degrees of freedom are fixed by prior information or knowledge) to find a more optimal solution, which is not limited to discretizing the multiple virtual images generated.

[0175] Embodiments are illustrated and described in detail in the drawings and the foregoing description, but such examples and descriptions should be considered specific examples or illustrations rather than limitations, and the present invention is not limited to the disclosed embodiments.

[0176] Other variations of the disclosed embodiments can be understood and achieved by a person skilled in the art in practicing the claimed invention, based on the study of the drawings, disclosures, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude the plural. A single processor or other unit may perform the functions of several of the items described in the claims.

[0177] The mere fact that certain means are described in different dependent claims does not mean that combinations of these means cannot be used advantageously. [Explanation of Symbols]

[0178] 100 doctors 110 Control Input Devices 120 Control Input Devices 130 Augmented Reality Devices 140 monitors 200 X-ray imaging devices 210 X-ray source 220 X-ray detectors 230 movable arms 300 Movable Patient Tables 310 Patient Tables 320 movable arms 400 robotic devices 410 Robot Arm 420 Joint 430 End Effector

Claims

1. A system for intraoperative navigation, Equipped with a processing unit and tracking device for real-time or near-real-time tracking of surgical objects in a 3D coordinate system, The processing unit is configured to receive an X-ray image depicting an anatomical structure including at least one bone at a first time point, and to determine the position of the anatomical structure in the 3D coordinate system that constitutes an incomplete registration of the anatomical structure in the 3D coordinate system, the incomplete registration being determined based on a received 3D dataset of the anatomical structure and a plurality of virtual X-ray images generated from different directions based on the 3D dataset of the anatomical structure. Between the first and second time points, the processing unit is configured to provide navigation information based on the incomplete registration of the anatomical structure with respect to at least one of the following: (i) soft tissue incision; (ii) Providing a soft tissue gateway to at least one bone, (iii) Tracking the movement, registering it in the 3D coordinate system, and determining the end position of the surgical object tracked by the tracking device, wherein the surgical object is moving outside of or on the surface of the at least one bone. At the second time point, the processing unit is configured to determine the complete registration of the position of the anatomical structure in the 3D coordinate system, the complete registration being determined based on the incomplete registration and at least one of the following: (a) Additional X-ray images, (b) A video output of the surgical object which is an endoscope, wherein the video output of the endoscope depicts the region of the anatomical structure, and the video output is processed and compared with the 3D dataset, (c) A point in the surgical object, wherein the end position of the movement of the surgical object is at a predetermined position in the at least one bone, (d) A point in the surgical object, wherein the end position of the movement of the surgical object is a contact point of the surgical object that contacts the surface of the at least one bone, (e) a system comprising at least one of the following: the contact point or at least a partial segmentation of the at least one bone adjacent to the predetermined location.

2. The system according to claim 1, wherein at least a portion of the surgical object is depicted in the X-ray image, the geometric shape of at least a portion of the surgical object is known, and the incomplete registration includes determining at least three degrees of freedom of the position of the surgical object in the 3D coordinate system.

3. The system according to claim 1, wherein a reference object is attached to the at least one bone, at least a portion of the reference object is depicted in the X-ray image, the geometric shape of the at least portion of the reference object is known, and the incomplete registration includes determining at least three degrees of freedom of the position of the reference object in the 3D coordinate system.

4. The system according to any one of claims 1 to 3, further comprising a robotic device for treating a patient, wherein a surgical object is attached to the robotic device and the processing unit is configured to control the operation of the robotic device.

5. The system according to claim 4, wherein the robotic device comprises a robotic arm configured to hold the surgical object for treating the patient, and the controlled motion of the robotic device is the motion of the robotic arm together with the surgical object at least one degree of freedom.

6. The system according to claim 5, further comprising an input device, wherein the input of the input device defines the movement of the robot arm.

7. The system according to claim 6, which enables a surgeon to operate on the patient remotely by combining automated robotic steering provided by the intraoperative navigation with manual robotic steering to cover all surgical operations, and further enables the surgeon to be outside the radiation field of the imaging device.

8. The system according to any one of claims 1 to 7, further comprising a plurality of cameras, wherein at least one of the cameras is mounted on a robotic arm controlled by the system.

9. The system according to any one of claims 1 to 8, wherein the processing unit provides the calculated position of the robotic device, and the system further comprises a display for visualizing the calculated position.

10. The system according to claim 9, wherein the calculated position is visualized before the robotic device is controlled to change its position toward the calculated position.

11. The system according to claim 9 or 10, further comprising an augmented reality function with the aforementioned display.

12. The system according to claim 4, wherein the operation of the robotic device includes soft tissue treatment of the patient, the soft tissue treatment is controlled by the system or, in the case of manual operation of the robotic device, is limited by the system, and the soft tissue treatment of the patient includes at least one of the group consisting of removal of at least a portion of the intervertebral disc, performing an incision, treating the dura mater, treating the spinal cord, treating nerves, and treating blood vessels.

13. The system according to claim 4, wherein the operation of the robotic device includes bone treatment of the patient, the bone treatment is controlled or limited by the system, and the treatment of the patient's bone includes at least one of the group consisting of drilling, excising, repositioning, inserting an implant, connecting an implant, removing an implant, inserting a bone graft, inserting a cage for a bone graft, inserting an artificial intervertebral disc, and performing an osteotomy.

14. The system according to any one of claims 1 to 13, wherein image features are identified in the received X-ray image, image features are identified in the virtual X-ray image, and the processing unit is further configured to perform a comparison between the image features in the virtual X-ray image of the plurality of virtual X-ray images and the image features in the received X-ray image by applying at least one of the group consisting of image rotation, image zoom, and translation in the image plane to determine one of the plurality of virtual X-ray images which best matches the image features of the image features in the received X-ray image, and the determined virtual X-ray image of the plurality of virtual X-ray images is used for the incomplete registration of the anatomical structure in the 3D coordinate system.

15. The aforementioned processing unit is The method involves generating a second set of virtual X-ray images, wherein the virtual X-ray images of the second set of X-ray images are generated as projection images of the 3D dataset of the anatomical structure, and the distribution of different imaging directions is near the imaging direction of the determined virtual X-ray images. Identifying image features within the second set of virtual X-ray images, The system according to claim 14, wherein a comparison is performed between the image features of the virtual X-ray images of the second plurality of virtual X-ray images and the image features of the X-ray image by applying at least one of the group consisting of image rotation, image zoom, and translation in the image plane, in order to determine one of the virtual X-ray images of the second plurality of virtual X-ray images that best matches the image features of the X-ray image, the determined virtual X-ray image of the second plurality of virtual X-ray images is used to determine the complete registration of the position of the anatomical structure.

Images