System and method for mixed reality-assisted 3D navigation for musculoskeletal surgery based on X-ray images while the patient is moving.

Abstract

This invention provides a system and method for providing information about surgical objects and anatomical structures based on CT data, 3D X-ray scans, and X-ray images. [Solution] The X-ray image is a 2D projection image that defines a 3D image coordinate system, generated in the imaging direction. The 2D projection image may depict some of the patient's anatomical structures and at least some of the surgical objects. To determine the spatial relationship between the 3D image coordinate system and the 3D data coordinate system, some of the patient's anatomical structures are located based on the X-ray image. Subsequently, the 3D position of the surgical object relative to the region of interest is determined in the 3D data coordinate system based on information regarding the determined spatial relationship between the first 3D image coordinate system and the 3D data coordinate system, the spatial relationship between the surgical object and the first 3D image coordinate system, and the geometric aspect.

Description

Technical Field

[0001] The present invention relates to the fields of computers and robot-assisted surgery. Further, the present invention relates to systems and methods for providing information regarding surgical objects and anatomical structures based on CT data or another type of 3D X-ray scan, and X-ray images. In particular, the present invention relates to systems and methods for automatically determining the spatial position and orientation of a surgical object relative to a patient while the patient is moving. The method can be implemented as a computer program executable on a processing unit of the system.

Background Art

[0002] Spinal surgery, particularly spinal fixation, is a commonly performed surgical procedure. However, surgery on the spine also poses a significant risk to the patient. An incorrect perforation performed on the spine can damage the spinal cord, which can cause serious damage to the nerves or the spinal cord's covering, leading to chronic pain, permanent paralysis, incontinence, or sexual dysfunction. To reduce these risks associated with incorrect perforations, computer-assisted navigation is already used to some extent in spinal surgery. Computer assistance involves guiding the surgeon, ensuring that the perforation is performed in the correct location, and that the pedicle screws are properly positioned, etc. This involves determining the accurate relative 3D position between the surgical tool (such as a drill) and the patient (e.g., relative to a desired perforation trajectory).

[0003] Most existing navigation systems require additional procedural steps and devices such as 3D cameras, trackers, and reference bodies. In navigation spinal surgery, most current systems use optical tracking, where a dynamic reference frame is attached to the spine and a reference body is attached to the surgical tool to track (any patient movement). In this case, all reference bodies need to be constantly visible to the 3D camera. Such an approach has a number of drawbacks including, but not limited to, the following. • The validity and accuracy of registrations must be continuously monitored. If the tracker moves, registration may need to be repeated. If the tracker's movement is not noticed, navigation instructions may become inaccurate, potentially harming the patient. • Accuracy decreases as the distance from the camera increases.

[0004] In summary, existing navigation systems are prone to errors. The long chain of errors in the navigation workflow—registration, instrument calibration, and real-time tracking—means that the accuracy of the navigation system can only be as good as the weakest part of the chain. Therefore, even when working with a navigation system, surgeons must always ensure the accuracy of navigation instructions through uncertain verification procedures such as tactile and landmark confirmation.

[0005] Robot-assisted surgical systems are becoming more widespread due to the perceived precision they offer. However, because robots operate primarily based on instructions provided by navigation systems, they can only be as accurate as the information provided by those navigation systems. Existing navigation systems are prone to errors, so current robotic surgeons cannot be trusted to autonomously perform any surgical procedure. Rather, robotic surgeons are simply automated holding arms that hold instruments. The actual perforation still needs to be performed by the surgeon.

[0006] The main problem with conventional navigation systems is that they do not determine the actual relative 3D position between surgical tools and anatomical structures, but rather infer their position and orientation by tracking externally attached reference objects (e.g., drills) using cameras. By their nature, these cameras can only see externally fixed reference objects and cannot see the drill itself, which is located inside the bone. If either of the reference objects moves, or if the drill bends inside the bone, the navigation system may not notice this fact, providing incorrect information and potentially harming the patient.

[0007] (i) It is desirable to have a navigation system that does not require any additional procedures or devices for navigation, and (ii) can reliably determine the actual 3D position and orientation of the surgical object (e.g., a tool or implant) relative to the patient. This must also reliably enable navigation during surgery on the thoracic spine that is moving after or while the patient or the patient's anatomical structures are moving, for example, due to the patient's breathing, or due to other patient movements, such as movements due to applied pressure that may occur during drilling procedures, such as distal locking of a long nail in a long bone or insertion of a screw such as a pedicle screw or pelvic screw.

[0008] Such navigation systems may be particularly useful in spinal surgery, but they can also be used in many other surgical procedures where the spatial relationships of surgical objects or implants to the patient must be precisely determined. Such navigation systems may even be used in combination with the conventional navigation systems described above to provide redundant navigation information, and thus make navigation very robust, and this level of safety may enable autonomous robotic surgery. [Overview of the Initiative]

[0009] At least one or the other of the above-mentioned problems is mitigated or resolved by the subject matter of each independent claim. Further embodiments are described in the respective dependent claims.

[0010] A system and method are proposed for determining the 3D spatial relationship or 3D position of surgical objects to a portion of a patient's anatomical structure, without the need for reference bodies or trackers.

[0011] Spatial relationships can be understood as constraints on parameterization by at least one degree of freedom to partially or completely describe the position and orientation between two 3D coordinate systems (e.g., (1) a reference system for surgical objects and (2) a 3D coordinate system for X-ray images). Correspondingly, the position and orientation between an object and a coordinate system can be partially or completely described.

[0012] For example, two, three, four, or five degrees of freedom of a surgical object may be determined within the image's 3D coordinate system. Based solely on a single 2D projection image, the spatial position of an object in the image's 3D coordinate system may be determined by, for example, five degrees of freedom. In the context of this disclosure, this is referred to as “position estimation,” meaning that the position in the X-ray orientation is determined with significantly less precision than its 2D position in the image and its three additional degrees of freedom (DOF) of rotation. Furthermore, some components of the 3D orientation may also exhibit reduced precision or be completely immeasurable, yet position estimation is still performed. A cylindrical object may be described by two precise positional degrees of freedom and one precise rotational degree of freedom, with the other degrees of freedom being less precise or even undetermined.

[0013] Throughout this disclosure, the term “imaging direction” (also called “line of sight direction”) on an object (or region of interest) means the 3D angle at which the X-ray beam passes through a selected point on the object (or region of interest). For example, the imaging direction on a region of interest represents the orientation of the X-ray machine relative to anatomical structures within the region of interest. The imaging direction may be determined as part of the position estimation of the object (e.g., the anatomical structures of a patient), which emphasizes that this is a spatial relationship between the object (e.g., region of interest) and the X-ray image 3D coordinate system.

[0014] Based on a single projection image, it may be impossible to reliably determine the tilt rotation of an object in the direction of image depth because the object is not fully visible in the image (this is typically always the case with a drill bit), and its imaging depth cannot be accurately determined. In this case, determining 4DOF may be sufficient as an intermediate step to determining 5DOF or even 6DOF, based on additional steps such as receiving information about the 3D position of anchor points in a coordinate system, such as the 3D coordinate system of another object of interest, such as the anatomical structures of a patient. This may make it possible to determine its 3D position and its tilt relative to other objects.

[0015] In navigation tasks, it may be perfectly sufficient to determine the 3DOF (3 degrees of freedom) in translation and the 2DOF (2 degrees of freedom) in rotation. This is especially true when determining the relative 3D position of a drill bit to the anatomical structures of a patient, for example. In this case, determining the rotational position around the drill axis may not be relevant.

[0016] 6DOF adequately describes the position and rotation of any stationary object. Some objects, such as Kirschner wire or drill bits, exhibit deformations such as bending, and therefore cannot be fully described with 6DOF; they can only be described with more degrees of freedom, e.g., 8DOF. In such cases, it may still be sufficient to focus on a subset of degrees of freedom that adequately describes the part of the object of interest. For example, the tip position of a curved drill bit could be described with 3DOF, or the area around the tip could be estimated with 4DOF.

[0017] The proposed system and method (to be implemented as a computer program product) is configured to determine 3D spatial relationships based on an X-ray image that depicts (at least partially) at least one surgical object and a portion of the patient's anatomical structure, in order to determine the 3D position of at least one surgical object relative to the patient's anatomical structure at a given time.

[0018] Furthermore, real-time tracking of the tool may be provided (without requiring any additional trackers or reference bodies attached to the patient's anatomical structure) to determine the 3D position of at least one surgical object relative to the patient's anatomical structure at a different point in time. This may also be the case when the patient's anatomical structure (region of interest) is moving.

[0019] Generally, a system is proposed for determining the 3D position of a surgical object relative to a region of interest within a patient's anatomical structure. The system comprises a processing unit and a computer program product containing instructions executable on the processing unit. The computer program product can execute a computer implementation method, which includes the step of receiving an X-ray image and a 3D dataset. The X-ray image is a 2D projection image that defines a 3D image coordinate system, generated in the imaging direction. When the method is applied, the 2D projection image may depict a portion of the patient's anatomical structure and at least a portion of the surgical object. The received 3D dataset describes a portion of the patient's anatomical structure and the region of interest in a 3D data coordinate system.

[0020] To determine the spatial relationship between the 3D image coordinate system and the 3D data coordinate system, the position of some of the patient's anatomical structures is estimated based on the X-ray images.

[0021] Next, the 3D position of the surgical object relative to the region of interest is determined in the 3D data coordinate system based on (I) the determined spatial relationship between the first 3D image coordinate system and the 3D data coordinate system, (II) the spatial relationship between the surgical object and the first 3D image coordinate system, and (III) information regarding the geometric aspect.

[0022] The spatial relationship between the surgical object and the 3D image coordinate system is derived by at least one of the following: (A) inferring a neural network; (B) utilizing multiple virtually generated X-ray images, the images depicting parts of the surgical object from various imaging directions; and (C) applying an algorithm using a 3D model representing at least a portion of the surgical object.

[0023] Information relating to a geometric aspect includes (a) that the geometric aspect of a surgical object is located near the surface of at least a portion of the region of interest; (b) points, axes, or planes in a first 3D image coordinate system that constrain the possible choices regarding the spatial relationships of the geometric aspect; and (c) one or more 3D positions of a target point in a 3D data coordinate system that is close to the geometric aspect, such that the projection of the target point in a 2D projected image and the projection of the geometric aspect in a 2D projected image are close to each other.

[0024] The embodiments of this disclosure may apply, but are not limited to, to open surgery, minimally invasive surgery, endoscopic surgery, and percutaneous spinal surgery in the treatment of degeneration, malformation, tumor, trauma, or infection, including cervical and thoracolumbar discectomy, decompression procedures, motion-preserving surgery, stabilization procedures such as vertebral ossoplasty and kyphosis plasty, spinal fusion such as ACDF, TLIF, PLIF, LLIF, OLIF, XLIF, open surgery, minimally invasive or percutaneous pedicle screw surgery, tumor debulking, or vertebral resection.

[0025] A surgical object may be a surgical instrument or tool, such as a drill, needle, or chisel, or it may be an implant, such as a pedicle screw, nail, or plate. The surgical object has a known geometric shape, which is described by a 3D model. The 3D model may be a detailed abstraction of the object, describing, for example, its surface and density, or a less detailed abstraction, including, for example, length, diameter, or partial surface description.

[0026] Surgical objects may also have a point referred to as an "anchor point," the location of which may be known in the 3D model of the surgical object and may be identifiable in an X-ray image. An anchor point could be, for example, the tip of a drill. In the case of a chisel, the anchor point could be one edge or midpoint of the chisel blade. An anchor point can be more abstract, such as the center of a cylindrical surgical tool that is not necessarily part of the object, or an axis passing through the center of the object.

[0027] A 3D data set describing a part of a patient's anatomical structure can be generated by a computed tomography (CT) scan or some other type of 3D X-ray scan, which can be acquired preoperatively or intraoperatively. The 3D data set includes a region of interest that needs to be sufficiently rigid. The region of interest can be, for example, a volume including bone fragments, a specific vertebra, or two adjacent vertebrae, which can be assumed to be sufficiently rigid. The region of interest can also be the entire 3D data set. To accurately describe at least the patient's anatomical structure within the region of interest, a one-to-one correspondence between points in the physical world and points in the 3D data set can be established at least within the region of interest. Thus, the 3D data set can define a 3D coordinate system within the physical 3D space. This can be for the purpose of determining the 3D position of a surgical object in this 3D coordinate system, or in other words, determining the 3D position of a surgical object (in the physical world) relative to the region of interest within the patient's anatomical structure.

[0028] According to one embodiment, in preoperative or intraoperative planning, target points within the 3D data set and within the 3D data set of the patient's anatomical structure are determined. The target points can be determined preoperatively or intraoperatively, either manually by a surgeon, automatically by a system, or semi-automatically by a surgeon with some automated assistance by the system. Thus, the position of the target points in the 3D coordinate system defined by the 3D data set is known. The target points can be selected such that the anchor points of the surgical object can be positioned (or near) on the physical location of the target points in the patient, which can be achieved, for example, by selecting target points on the bone surface.

[0029] In another embodiment, the target point may be the starting point of the target path, and its end point will be referred to as the target end point. The target path is within the region of interest, and its position and orientation within the 3D data set can be determined manually by the surgeon, automatically by the system, or semi-automatically by the surgeon with some automated assistance by the system during preoperative or intraoperative planning. Thus, when the target path is available, its position and orientation in the 3D coordinate system are known. For example, when treating a vertebra, the target path may correspond to a planned drilling trajectory (typically a straight line), and the target point is located on the surface of the vertebra.

[0030] The method disclosed herein teaches a method for determining the 3D position of at least one surgical object in a 3D coordinate system defined by a 3D data set describing the patient's anatomical structure. By doing so, the relative 3D position of the surgical object with respect to the region of interest of the patient's anatomical structure is determined. Thus, if the region of interest includes the planned target path, the relative 3D position of the surgical object with respect to the target path within the patient's anatomical structure can also be determined. It may also be an objective to provide instructions for aligning the surgical object with the target path (if available), for example, for drilling along the target path.

[0031] Note that some surgical objects, such as drill bits, have rotational symmetry about an axis. In the case of such objects, since rotation about the axis of the object is irrelevant, it is sufficient to determine five degrees of freedom of their 3D positions with respect to the patient's anatomical structure. For example, in the case of a drill bit, it is sufficient to determine the 3D position of the drill tip and the 3D orientation of the drill axis.

[0032] In the context of this disclosure, the rotation of a surgical object about its axis may be irrelevant to the 3D spatial relationship of the surgical object with respect to anatomical structures. In other words, the 3D position is considered to be determined regardless of whether the actual rotation angle of the surgical object about its axis of rotation is precisely determined or not.

[0033] Neither a target point nor a target path is necessarily required. In one embodiment, it is possible to work using only anchor points (e.g., the aspect of a geometric anchor, such as the tip of a drill). In this case, the surgeon may decide during the surgery whether the 3D position and orientation of the surgical object relative to the patient's anatomical structure is satisfactory, and if not, how to correct it.

[0034] It is emphasized that 3D datasets do not necessarily need to be segmented. In particular, it is not necessary to identify bone surfaces within the 3D dataset (other than selecting target points on the bone surface). Furthermore, it is not necessary to know, or explicitly determine, the exact location and shape of the region of interest within the 3D dataset. For example, when treating a vertebra, a CT scan may include not only this vertebra but also several adjacent vertebrae (and even surrounding soft tissue). Initially, the vertebra does not need to be identified within the 3D dataset. A rough approximation of the region of interest within the 3D dataset can be derived from the target points. An anchor point defines the 2D region of interest in the X-ray image. For subsequent calculations, data points within the 2D region of interest may be weighted, with points further away from the anchor point receiving a lower weight.

[0035] The location estimation of a 3D region of interest depicted in an X-ray image can be determined by computing a digitally reconstructed X-ray image (DRR) for multiple location estimations from a 3D dataset. Alternatively, a model of the imaged structure may be used to determine the location estimation. Location estimation may also be determined based on location and / or motion sensors provided within the imaging device. Location estimation can also be determined by starting from a previous registration of 3D data and a 3D image coordinate system, tracking the imaging device using an arbitrary tracking device, such as a mixed reality (XR) device, and adding the detected relative motion of the imaging device to the previous location estimation from the previous registration.

[0036] When determining the location by acquiring the DRR that best matches the X-ray image, it may also be possible to use a numerical optimizer to find the optimal DRR. The DRR does not need to be limited to the region of interest. When evaluating the best match between the DRR and the X-ray, the region of interest may be emphasized (e.g., by appropriate weighting). If the acquired X-ray image is distorted (i.e., if the X-ray image was acquired using an image intensifier rather than a flat panel receiver), the distortion may be stronger as the distance between points increases, so it may be necessary to use a smaller area in the X-ray image for DRR matching (or emphasize points closer to the anchor point). Since distortion typically affects areas further away from the central X-ray beam more strongly, it may be useful to ensure that the anchor point is close to the center of the X-ray image.

[0037] As described in detail in U.S. Patent Application Publication No. 2021 / 0248779, the 3D position of thin surgical objects, such as drill bits, is not always uniquely determined based on a single X-ray image. Therefore, generally, when only one X-ray image is available, there may be ambiguity in the relative 3D spatial relationship between the surgical object and the patient's anatomical structure. This disclosure teaches how to resolve such ambiguity by first determining the 3D position of anchor points in a 3D coordinate system defined by a 3D dataset. Various methods for achieving this are disclosed.

[0038] A method according to one embodiment that does not use a target point requires that the anchor point be located on the bone surface. Based on a single X-ray image, the 2D position of the anchor point in the X-ray image can be determined, and the imaging direction of the X-ray image onto the region of interest can be determined from DRR matching. In 3D space, the point on the anchor point where the virtual projection into the X-ray image lies determines the line. Since the anchor point, which is a geometric anchor aspect, lies on the bone surface, the position of the anchor point in 3D can be found by determining the point on the line that lies on the bone surface. It is sufficient to determine the location on the bone surface along the line. Segmentation of the entire bone surface is not necessary. It is understood that "on the bone surface" or "on the bone surface" also includes the fact that it is sufficient for the anchor point to be in the vicinity of the bone surface, for example, less than 2.5 mm or less than 1.5 mm from the bone surface. Alternatively, the exact distance of the anchor aspect from the bone surface may be known. In this case, this distance can be taken into consideration.

[0039] Another method involves establishing a correspondence between anchor points (on surgical objects) and target points (which may be geometric aspects of a 3D dataset). This is done by virtually projecting the target points into the X-ray image, for example, by utilizing known imaging directions on the region of interest determined by DRR matching.

[0040] If the location of the anchor point coincides with the location of the target point in the X-ray image, it can be assumed that the 3D position of the anchor point coincides with the 3D position of the target point in physical space. This is because the target point is selected so that the anchor point of the surgical tool can be positioned on the target point (in physical space). Of the three coordinates of the 3D position, two degrees of freedom are determined by the X-ray image, and the third degree of freedom is determined by the a priori information that the target point and the anchor point are on or at a defined distance from the bone surface.

[0041] For example, if the anchor point is not on the bone surface because drilling into the pedicle has already begun, yet another method for determining the 3D position of the anchor point in a 3D coordinate system defined by the 3D dataset is to use additional X-ray images from different imaging directions. These additional X-ray images may be acquired at an earlier time, simultaneously with the first X-ray image at the first time (e.g., if a G-arm is used), or at a later time (e.g., if a robot capable of holding the surgical object stationary is used). The additional X-ray images depict at least a portion of the surgical object and the region of interest within the patient's anatomical structure. The 2D position of the anchor point in the additional X-ray images is determined. The imaging direction of the additional X-ray images onto the region of interest is determined, thereby allowing the additional X-ray images to be registered to the first X-ray images. This method assumes that there was no movement of the anchor point relative to the anatomical structure between the generation of the two X-ray images. The advantage of this method is that preoperative planning (e.g., defining the drilling trajectory and / or target point) is unnecessary. Furthermore, if the anchor point (e.g., the tip of the drill) is located below the bone surface, the possibility of undesirable slippage of the surgical tool is minimized.

[0042] The anchor point may also be a geometric aspect of a surgical tool, such as the tip and central axis of a drill bit. 3D position estimation can be achieved if the anchor point can be partially estimated in the X-ray image at a first time point, or at a later time point. For example, the central axis of a drill bit may be identifiable in the X-ray image at a first time point, while only the tip of the drill bit may be identifiable in the X-ray image at a later time point. Assuming the drill tip was not moved, it is still sufficient to estimate the 3D position of the anchor point in a 3D coordinate system. Given the imaging direction of the X-ray onto the drill bit at both the first and later time points, the detected tip and central axis in the image provide a geometric aspect that can be transformed from the image 3D coordinate system to a 3D coordinate system. The tip of the drill bit at a later time point represents a line in the 3D coordinate system, and the central axis at the first time point represents a plane in the 3D coordinate system. The intersection of the line and the plane provides an estimate of the 3D position of the anchor point. In this case, the 3D orientation of the central axis of the anchor point is only partially estimated, because a more accurate estimation might require identifying the central axis in both X-ray images.

[0043] Estimation of the 3D position of an anchor aspect, which can be a geometric aspect, can also be achieved in a similar manner even when the surgical tool is moved. In the example of a drill bit, the anchor aspect of the surgical tool can be its central axis direction and its tip position. The drill bit can be positioned in the 3D coordinate system of an anatomical structure, for example, within bone. A first X-ray can be acquired at a first time point, in which the drill bit can be estimated to be positioned such that its tip is determined in 2D coordinates (at least 2DOF). The first X-ray image corresponds to a 3D image coordinate system, and the tip position estimation corresponds to an uncertain line between the X-ray source and the tip projected into the 2D image plane. At a second time point, another X-ray can be acquired, in which case the registration between the 3D image coordinate system and the 3D coordinate system of the anatomical structure is received or determined.

[0044] At the second time point, the drill bit may have been further inserted, although its tip is not in the same 3D position (relative to the anatomical structural coordinate system) as at the first time point, but the tip at the first time point is still close to the axis of the drill bit at the second time point. The drill bit can be positioned in the second X-ray image such that its tip and axis are determined in 2D (at least 3DOF), with the tip corresponding to an uncertain line in the second 3D image coordinate system and the axis corresponding to a plane in the second 3D coordinate system.

[0045] Using registration between coordinate systems, uncertain lines from the first 3D image coordinate system, as well as uncertain lines and planes from the second 3D image coordinate system, may be transformed into the 3D coordinate system of the anatomical structure. In this 3D coordinate system, the intersection of the uncertain plane of the second X-ray image and the uncertain line from the first X-ray image is the 3D point on the axis of the drill bit at the second time point.

[0046] To fully determine the 3D position of the drill tip and drill axis, it is necessary to constrain another degree of freedom, which may have several origins. For example, (i) the angle of the drill bit may be determined from the first X-ray (i.e., at least 3DOF) if, for example, the direction remains the same between the first and second X-rays; (ii) the drill bit position estimation in the second X-ray may provide a more detailed estimate regarding rotation (i.e., at least 4DOF); and (iii) there may be reasonable assumptions regarding a fourth degree of freedom. This constraint allows the axis of the drill bit at the second time point to be determined. Given the axis of the drill bit, the tip point of the drill bit at the second time point can be determined as the intersection of the axis of the drill bit and its indeterminate line from the second X-ray image. Given the tip position and axial direction of the drill bit (i.e., the 3D position of the anchor aspect), 5DOF can be determined for the second time point. The rotation of the drill bit around its main body can also be determined as part of the position estimation in the second X-ray image.

[0047] The geometric aspect of an object is a descriptive attribute. Examples of geometric aspects include a subset of the surface of a drill bit, feature points at the tip, the mass centroid of a bone surface, the projected trajectory of a drill bit, and the curved central axis of a curved cylindrical surgical tool. Depending on the geometric aspect, less than 6DOF may be required to fully describe it with respect to a 3D coordinate system. The tip of a drill bit can be fully described by its position, which has 3DOF, and the line requires only 5DOF. In the case of a curved cylindrical surgical instrument, the central axis may have more than 6DOF to fully describe it, but it may be sufficient to reduce the set of parameters and only partially estimate its properties.

[0048] Typically, a 3D dataset (e.g., a CT scan) contains scaling information (i.e., size information) for the anatomical structures described within the dataset. Furthermore, the scale of the scene depicted in the X-ray can also be provided by depicted surgical objects whose precise geometric shapes are known, or by other surgical objects with known geometric shapes that can be depicted in the X-ray image (e.g., already implanted pedicle screws). This redundancy regarding scale information can be used to distinguish between different surgical objects in the X-ray image when they have the same geometric shape but different sizes (e.g., drills with different diameters) by comparing the scale of the 3D data with the size of the surgical objects. If the acquired X-ray image is distorted, the same information (i.e., the 3D dataset, a 3D model of the surgical object, or another surgical object with a known geometric shape) may be used to correct the distortion.

[0049] If the location of the anchor point does not coincide with the location of the (virtually projected) target point in the X-ray image, the system may provide instructions on how to move the surgical tool so that the anchor point is closer to the target point, after which a new X-ray can be acquired. If the location of the anchor point is in the vicinity of the target point in the X-ray image, and a local model of the bone surface in the vicinity of the target point is available, the 3D position of the anchor point can be obtained from the local model for the 2D position of the anchor point detected in the X-ray image. This is sufficient to determine the 3D relative spatial relationship between the surgical object and the patient's anatomical structure. Such a local model of the bone surface could be, for example, the median shape of the bone, a first-order approximation of the bone shape, or a local segmentation of the bone surface in the vicinity of the target point. The correct model may be selected based on labeling of the 3D dataset (e.g., classifying vertebrae), which may be done automatically, semi-automatically, or manually. The size of the target point's neighborhood may be fitted based on the local model; for example, the less variation there is in the local model near the target point, the larger the neighborhood that can be selected. Naturally, if segmentation of the 3D dataset (i.e., a model of the entire bone surface) is available, this can be taken into account, which may improve accuracy.

[0050] The 3D position of an anchor aspect may have significant errors in some dimensions, but is sufficiently accurate in the dimension being adjusted, for example, when navigating a surgical tool to a target point. When navigating an anchor aspect to a target point, the navigation information may be constrained to 2D translation (e.g., the surface of a bone), but the third dimension of translation can only be estimated with greater error. This is particularly useful when 2D navigation may be estimated based on an X-ray image, and the 2D navigation direction is not perpendicular to the projection plane of the X-ray image.

[0051] Alternatively or additionally, the location estimation of the anchor aspect can utilize a target point to constrain the 3D coordinates. For example, the drill tip may be estimated in 2D within the X-ray image, while the position on the uncertain line (i.e., the line from the X-ray source to the projected anchor aspect in the image plane) can be assumed to be close to the target point, assuming the target point is estimated or known within the 3D image coordinate system.

[0052] Alternatively or additionally, the system may include means for tracking the anchor aspect from the time the X-ray was acquired to any point later, utilizing the estimated position of the anchor aspect in a 3D image coordinate system relative to the target point. Tracking may be performed by external sensors, such as a camera or by command or tracking from sensors of a surgical robot. Based on this tracking, the direction and distance to the target point may be updated and provided to the surgeon or robot in real time (or near real time).

[0053] It will be understood that the target point needs to be linked to an estimate of the anchor aspect in order to provide navigation instructions. For example, the target point can be derived from a 3D dataset, where the 3D coordinate system of the 3D dataset is registered in the 3D image coordinate system so that the target point can be mapped to the 3D image coordinate system.

[0054] As mentioned above, it is not necessary to know the exact location and shape of the 3D region of interest within the 3D dataset. Furthermore, it is not necessary to establish a precise correspondence between the 2D region of interest in the X-ray image and the 3D region of interest in the 3D dataset. However, there may be cases where several possible 3D regions of interest exist. For example, in spinal surgery, since multiple vertebrae are treated, multiple 3D regions of interest may exist. In such cases, a human surgeon can manually identify which 3D region of interest should be treated. It may also be possible to automatically select the relevant 3D region of interest corresponding to the anchor point indicated by the surgical tool by performing DRR matching for each possible 3D region of interest and selecting the best such match.

[0055] The region of interest is an approximation of an area (or volume) that can be assumed to be sufficiently rigid, thereby allowing the 3D dataset to describe the patient's anatomical structures within the region of interest with sufficient accuracy. For example, because the spine is flexible, individual vertebrae may move relative to one another. If the patient is moved after the 3D dataset has been acquired, some relative movement of individual vertebrae may need to be predicted. This means that the area that can be assumed to be rigid may be larger in intraoperative 3D X-rays (without subsequent patient movement) than in preoperative CT scans (with subsequent patient movement), assuming no significant movement due to respiration.

[0056] 3D datasets may be input into the system or derived by the system. For example, the system may estimate 3D data based on X-rays using a statistical model of relevant parts of an anatomical structure. As an example, in the context of spinal surgery, the statistical model (e.g., an active shape model) may describe the surface or density of one or more vertebrae. The 3D dataset may also be updated or extended with further X-ray images to improve accuracy and reliability.

[0057] To improve the accuracy of the derived 3D dataset, the system can utilize the geometric aspects of surgical objects by registering aspects between two or more images. For example, a drill bit may be positioned on bone in the first and second X-ray images, with the drill axis changing between images, but the tip remaining the same. Similarly, a screw may be identified in both X-ray images to constrain the possible field of view directions of the X-ray images. This can be used to determine or constrain the registration of the 3D image coordinate system, thus improving its accuracy.

[0058] Alternatively or additionally, prior knowledge of the geometric aspects of surgical tools may be utilized. For example, if the drill tip is positioned on the surface of a bone, the modeled bone surface needs to be close to the detected drill tip position. Using drill tip position estimation can constrain the system when estimating 3D data, thus improving the accuracy of the estimation.

[0059] The methods taught herein may complement existing navigation techniques. Another aspect may be the continuous incorporation of information from intraoperative radiography. The systems and methods disclosed herein do not require a camera or other sensor for navigation, but may nevertheless be combined with a camera or other sensor (e.g., a sensor mounted on a robot) for navigation. Information from radiographic images and information from cameras or other sensors may be combined to improve the accuracy of determining relative spatial position and orientation, or to resolve any remaining ambiguities (which may be, for example, due to occlusion). If the tool is held by a robot or robotic arm, information provided by the robot or robotic arm itself (e.g., information about the movements it has performed) may also be considered.

[0060] The methods taught in this disclosure can also be combined with extended reality devices such as head-mounted augmented or mixed reality glasses. For example, by tracking a power tool with a head-mounted extended reality device, changes in the 3D position and orientation of a drill bit inserted into the power tool between two points in time can be determined. Thus, if the 3D position of a surgical object (such as a drill bit) relative to an anatomical structure is known at a first point in time, the 3D position of the surgical object relative to the anatomical structure can be determined at a second point in time by tracking the power tool using an extended reality device. If only the tilt of the surgical object is known to have changed between the two points in time (e.g., the power tool is only tilted, and the drill tip remains in place), this knowledge can be used by the system.

[0061] It should be noted that information regarding the change in the 3D position of a surgical object between two points in time may also be provided by the robotic arm holding the surgical object, since the robotic arm can typically track its movement in 3D space.

[0062] Possible applications of this disclosure include, for example, insertion of screws into the pedicle, wedge osteotomy, screws into the sacroiliac joint, screws connecting two vertebrae, screws through the pelvis, screws through the acetabulum, or any type of bone drilling for perforation of the cruciate ligament. The disclosed teachings may be used, for example, for perforation, reaming, milling, chiseling, sawing, resection, and implant positioning, and therefore may also assist, for example, osteotomy, laminectomy, tumor resection, and total hip arthroplasty.

[0063] It will be understood that a system may comprise a processing unit, and a method may be implemented as a computer program product that can be executed on that processing unit.

[0064] According to one embodiment, a system and method can cause an imaging device to generate an X-ray image or some type of 3D X-ray scan (e.g., a CT scan). Additionally or alternatively, the system and method can control the imaging device to move it to a new position for generating an X-ray image from a different imaging direction. Such a different imaging direction may be a suitable imaging direction proposed by the system. The current imaging direction may be determined based on the movement and / or position sensors inside the imaging device.

[0065] It should be noted that the image data of the processed X-ray images can be received from an imaging device, such as a C-arm, G-arm, or two-plane 2D X-ray device, or directly from a database. A two-plane 2D X-ray device has two X-ray sources and a receiver offset by any angle.

[0066] According to one embodiment, objects in an X-ray image can be automatically classified and identified within the X-ray projection image. However, objects may also be manually classified and / or identified within the X-ray projection image. Such classification or identification can be assisted by the device by automatically referencing structures recognized by the device.

[0067] It should be noted that the processing unit may be implemented by a single processor that performs all steps of the processing, or by a group or multiple processors that do not need to be located in the same place. Cloud computing, for example, allows processors to be located anywhere. For example, the processing unit may be divided into a first subprocessor that controls user interaction, including a monitor for visualizing the results, and a second subprocessor (which may be located elsewhere) that performs all the calculations. The first or other subprocessor may also control the movement of the C-arm or G-arm of an X-ray imaging device, for example.

[0068] According to one embodiment, the device may further include storage means that provide a database for storing, for example, X-ray images. Such storage means may also be provided in a network to which the system may be connected, and it will be understood that data related to a neural network may be received through that network. Furthermore, the device may include an imaging unit for generating at least one 2D X-ray image, the imaging unit may be capable of generating images from various directions.

[0069] According to one embodiment, the system may include a device for providing information to a user, the information including at least one of a group consisting of X-ray images and instructions relating to steps of a procedure. It will be understood that such a device may be a monitor or augmented reality device for visualization of the information, or a speaker for providing the information acoustically. The device may further include input means for manually determining or selecting the location or part of an anatomical structure in an X-ray image, such as a bone contour, for example, to measure distance in the image. Such input means may be, for example, a computer keyboard, computer mouse, or touchscreen for controlling a pointing device, such as a cursor on a monitor screen that may be included in the device. The device may also include a camera or scanner for reading labels on packages or identifying surgical objects. The camera may also allow the user to communicate visually with the device by gesture or imitation, for example, by virtually touching the device displayed by virtual reality. The device may also include a microphone and / or speaker to communicate acoustically with the user.

[0070] According to one embodiment, the system may also comprise an extended reality device. The extended reality device will be understood to encompass all devices on the augmented reality spectrum and thus be a broader set of virtual reality, augmented reality, and mixed reality devices. This represents all devices that can be worn by a user and that (1) can alter the user's perception by any sense, including sight, hearing, smell, taste, and touch, and (2) can track its environment by various sensor information, including cameras, depth sensors, accelerometers, and magnetometers. These sensors may track an environment that includes imaging devices and / or at least one object, such as an anatomical model, surgical tool, or implant. It should be noted that the tracking information allows for the distinction between objects and / or devices. For example, the extended reality device may be head-mounted and comprise glasses, through which the user may view the environment, but the glasses are configured to visualize information or virtual representations of objects within the user's field of view.

[0071] It should be noted that all references to C-arm movement in this disclosure always refer to relative repositioning between the C-arm and the patient. Therefore, any translation or rotation of the C-arm can generally be replaced by the corresponding translation or rotation of the patient / OR table, or a combination of C-arm translation / rotation and patient / table translation / rotation. This may be particularly relevant when dealing with limbs, as it may actually be easier to move the patient's limbs than to move the C-arm. It should be noted that the required patient movement generally differs from the C-arm movement, and in particular, patient translation is not required, typically when the target structure is already in the desired position in the X-ray image. The system may calculate C-arm adjustments and / or patient adjustments. Furthermore, it should be noted that all references to the C-arm may also apply to the G-arm, O-arm, or similar.

[0072] The methods and techniques disclosed herein may be used in systems assisting a human user or surgeon, or in systems in which some or all of the steps are performed by a robot. Accordingly, references to “user” or “surgeon” in this patent disclosure may refer not only to a human user but also to a robotic surgeon, a mechanical assistance device, or similar apparatus. Similarly, whenever it is mentioned that instructions are given on how to adjust the C-arm, it should be understood that such adjustments may also be performed without human intervention, i.e., automatically, by the robotic C-arm, by the robotic table, or by an OR staff with some kind of automated support. It should be noted that because robotic surgeons and / or robotic C-arms can operate with greater precision than humans, repetitive procedures may require fewer repetitions, and more complex instructions (e.g., combining multiple repetitive steps) may be performed. A key difference between robots and human surgeons is the fact that a robot can keep the tool completely still between the acquisition of two X-ray images.

[0073] The computer program product may preferably be loaded into the random access memory of a data processor. Thus, a data processor or processing unit of a system according to one embodiment may be equipped to perform at least a portion of the described process. Furthermore, this disclosure may relate to computer-readable media, such as a CD-ROM, on which the disclosed computer program product may be stored. However, the computer program product may also be presented via a network such as the World Wide Web and downloaded from such a network into the random access memory of a data processor. Furthermore, the computer program product may also be executed on a cloud-based processor, and the results may be presented via a network.

[0074] It should be noted that prior information regarding surgical objects (e.g., drill bit size and type) can be obtained before or during surgery by simply scanning any markings on the packaging (e.g., barcode) or the surgical object itself.

[0075] As is evident from the above description, the primary aspect is the processing of X-ray image data, enabling the automatic interpretation of visible objects. The methods described herein should be understood as methods to assist in the surgical treatment of patients. Therefore, according to one embodiment, the methods do not necessarily include the step of surgical treatment of an animal or human body.

[0076] It will be understood that the steps of the methods described herein, in particular the steps of the methods described in relation to the workflow of embodiments, some of which are visualized in the figures, are major steps, and these major steps may be distinguished or divided into several substeps. Furthermore, there may be additional substeps between these major steps. It will also be understood that only a part of the whole method may constitute the present invention, i.e., steps may be omitted or combined.

[0077] It should be noted that the embodiments are described with reference to various subjects. In particular, some embodiments are described with reference to method-type claims (computer program products), and other embodiments are described with reference to apparatus-type claims (systems / devices). However, those skilled in the art will infer from the above and below descriptions that, unless otherwise specified, any combination of features belonging to one type of subject matter, and any combination of features relating to different subjects, are disclosed in this application.

[0078] The embodiments defined above, as well as further embodiments, features, and advantages of the present invention, may also be derived from the examples of embodiments described below and will be illustrated with reference to the examples of embodiments shown in the figures, but the present invention is not limited thereto.

[0079] The tip of an object, such as the tip of a sleeve, may or may not be depicted in a 2D X-ray image because it may be obscured by other parts of the same object or other objects. Nevertheless, it is possible to accurately detect a sufficient contour and potentially use the inner appearance of the sleeve in the X-ray image as well, and thus it may be possible to use this information to determine its relative 3D position and orientation to anatomical structures. This information can then be used as the basis for tracking the object by, for example, a camera and / or 3D sensor on a head-mounted display. By tracking the 3D position of the sleeve, the drilling trajectory can be determined and / or observed, but the drill or drilling machine itself does not need to be tracked at all. This can be advantageous because, for example, a portion of the drill bit outside the bone (and outside or in the absence of the sleeve) may be significantly bent, and this is especially true for long drill bits, so tracking the drilling machine to determine the actual drilling trajectory may result in inaccuracies or significant deviations between the drilling trajectory and the drilling machine's trajectory.

[0080] It may be advantageous to combine sleeve tracking to determine the drilling trajectory (i.e., determination of spatial relationships t at additional time points, e.g., real-time tracking) with tracking of the drilling machine to determine, for example, the drilling depth of the drill bit within the bone.

[0081] It may be advantageous to track the relative position of the drilling machine to the sleeve. This tracking may only need to be done in one dimension, for example, by tracking only the distance of the (calculated) position of the drill tip relative to the position of the sleeve tip in the direction of the drilling trajectory.

[0082] To ensure secure tracking of the sleeve, it may be shaped so that a sufficient portion of it is not obstructed by other objects such as instruments, puncture machines, or the surgeon's hand or arm.

[0083] Based on image processing of 2D X-ray images, it may be advantageous to have sufficient radiopaqueness of the sleeve, or a retaining arm spanning a sufficient volume that is securely attached to the sleeve, in order to ensure reliable detection and determination of the sleeve's 3D position relative to an anatomical structure, for example.

[0084] To enable reliable detection and image processing of a sufficient portion of the drill bit, it may be advantageous that the sleeve portion that actually covers the drill bit be transparent.

[0085] After incision, the procedure may begin with positioning / navigating the drill sleeve. As soon as the drill sleeve is positioned on the surface of the anatomical structure at the desired / planned entry point, a drill bit attached to an insertion tool (e.g., a power tool) may be inserted into the drill sleeve until it contacts the bone surface. A real-time tracking system (e.g., a head-mounted augmented reality device or robotic arm) may track the insertion tool and detect when it is on the surface. This detection can be made more reliable by, for example, rotating the insertion tool as soon as the tip of the drill bit reaches the surface. The drill sleeve will move with the patient's respiratory movements, as it may always remain at its anchor point on the surface of the anatomical structure. Any changes in the angle of the drill sleeve may be determined by a tracking system, which may be provided, for example, by a head-mounted display and its camera and / or 3D perception device. When drilling is initiated, tracking of the insertion tool and sleeve may allow for updating navigation information regarding the 3D position of the drill bit inside the bone / patient's anatomical structure.

[0086] In summary, a device may be provided for measuring drilling depth in a moving anatomical structure. The device may comprise a tracking system for tracking a surgical tool and a processing unit for processing information received from the tracking system, the surgical tool comprising at least a drill sleeve and a drill bit in a drilling machine, and the tracking system tracking the 3D position of the drill sleeve as it moves with the anatomical structure. Furthermore, the tracking system tracking the 3D position of the drill bit as it moves within the drill sleeve relative to the anatomical structure.

[0087] Based on the tracked 3D positions of the drill sleeve and the drill bit at multiple time points, the device's processing unit may be configured to determine the 3D position of the drill sleeve relative to the 3D position of the drill bit at each of the multiple time points, and as a result, to determine the maximum distance of the distal tip of the drill bit from the drill sleeve.

[0088] The maximum distance of the drill tip from the drill sleeve may correspond to the maximum drilling depth of the drill bit into the bone when the drill sleeve is in contact with the outer surface of the bone. [Brief explanation of the drawing]

[0089] [Figure 1] This shows 2D X-ray projection images of the spine from the anterior-posterior direction. [Figure 2] Figure 1 shows a 3D scan slice depicting a sagittal section of the same spine. [Figure 3] This shows a slice of the same 3D scan as Figure 2, depicting an axial section. [Figure 4] This illustrates an exemplary workflow for performing pedicle screw drilling based on a plan. [Figure 5] This provides a brief, exemplary workflow for performing pedicle screw drilling without changing the imaging direction, given prior knowledge of the bone surface. [Figure 6]This illustrates an exemplary workflow for performing pedicle screw drilling based on further X-ray images with different imaging orientations. [Modes for carrying out the invention]

[0090] Throughout the drawings, unless otherwise specified, the same reference numerals and numerals are used to indicate similar features, elements, components, or parts of the illustrated embodiments. Furthermore, the present disclosure will be described in detail with reference to the drawings, in relation to exemplary embodiments and not limited to the specific embodiments shown in the drawings.

[0091] This disclosure teaches a system and method for determining the 3D position and orientation of a surgical object relative to a portion of a patient's anatomical structure called a region of interest, which is described by a 3D dataset such as a CT scan or some other type of 3D X-ray scan, based on a single X-ray image. Without additional information, the 3D pose (i.e., 3D position and orientation) of a thin surgical object, such as a drill, cannot necessarily be uniquely determined based on a single X-ray image.

[0092] Figure 1 is a 2D X-ray projection image depicting the spine from the anterior-posterior direction. A surgical object (1.SO) is positioned so that its tip lies on the bone surface of vertebra T8 (1.T8). The two contours of the surgical object show two different 3D positions (1.P1 and 1.P2, respectively) and 3D orientations of the surgical object, whose contours (white solid lines labeled 1.O1 and 1.O2, respectively) are virtually identical. The tip of the surgical object in 3D space lies on the line defined by the drill tip and the X-ray source of the X-ray machine (typically the focal point), i.e., the epipolar line of the tip of the surgical object. Depending on the 3D position of the tip of the surgical object, the 3D orientation may be adjusted so that the projected contour of the surgical object fits its appearance in the 2D X-ray image. It should be emphasized that there are countless combinations of 3D positions and corresponding 3D orientations that lead to nearly identical contours of the surgical object, two of which are depicted in Figure 1.

[0093] Figure 2 shows a slice of the same 3D scan of the spine as in Figure 1, depicting a sagittal section. The white line (2.EL) indicates all possible 3D positions of the tip of the surgical object, corresponding to the epipolar line of the surgical object tip in Figure 1. Two points on the line have been selected, one (2.P1) where the tip of the surgical object is located on the bone surface of vertebra T8 (2.T8), and the other (2.P2) where it is located at a specific distance from the bone surface. Although the 3D orientation of the surgical object differs depending on its 3D position, it is clearly visible that both result in nearly identical contours in the X-ray projection image in Figure 1.

[0094] Figure 3 shows a slice of the same 3D scan as Figure 2, depicting an axial section. The white line (3.EL) indicates all possible 3D positions of the tip of the surgical object, corresponding to line 2.EL in Figure 2. As in Figure 2, two points on the line have been selected: one (3.P1) is where the tip of the surgical object is located on the bone surface of vertebra T8 (3.T8), and the other (3.P2) is where it is located at a specific distance from the bone surface. Here again, although the 3D orientation of the surgical object is different, it is clearly visible that both result in nearly identical contours in the X-ray projection image in Figure 1.

[0095] In other words, Figures 1-3 show two different 3D constellations with two different drill poses that result in identical (or nearly identical) X-ray projection images. The two drill constellations differ in imaging depth (the distance from the image plane, i.e., from the X-ray receiver) and inclination (i.e., 3D orientation). However, if it is possible to eliminate ambiguity regarding imaging depth (due to some priori information), the drill pose relative to the patient's anatomical structure can be uniquely determined.

[0096] This disclosure teaches how to eliminate ambiguity regarding imaging depth by utilizing prior information that the target point and anchor point are on or at a defined distance from the bone surface to establish a correspondence between the anchor point of a surgical object and the target point (whose position in a 3D coordinate system defined by a 3D dataset is known). This can be done based on a single X-ray image.

[0097] Alternatively, ambiguity regarding imaging depth can be eliminated without using any target points. According to one embodiment, the tip of the drill (i.e., the anchor point) is positioned on the bone surface near the intended drilling path (i.e., within the region of interest), and an X-ray image is acquired. The position of the drill tip is detected in the X-ray image. For the imaging direction to the region of interest, determined based on DRR matching of the 3D dataset to the X-ray image, a line (so-called epipolar line) in the 3D coordinate system defined by the 3D dataset is determined by the possible positions of the drill tip in 3D space where the virtual projection onto the X-ray image lies on the detected anchor point. Due to the prior knowledge that the drill tip is on the bone surface, the position of the drill tip in the coordinate system defined by the 3D dataset can be found by determining a point on the line that intersects the bone surface (i.e., is located on the bone surface). Finding the bone surface along the line can be done automatically, for example, by evaluating the Hunsfield values ​​in the 3D dataset along (or in the vicinity of) the line and searching for strong gradients. Naturally, if local segmentation of the bone surface within the relevant region is already available, this may be used to intersect the lines.

[0098] By acquiring additional X-ray images from the current and / or different imaging directions, it may be possible to improve the accuracy of determining the spatial relationship between the surgical object and the region of interest.

[0099] It is also possible to determine the 3D spatial relationship of the surgical object to the region of interest after the surgical object has been inserted into the patient, i.e., after the anchor point is no longer located on the bone surface. This is noteworthy, for example, when performing a drill. In such a case, an X-ray image is acquired at a first time point, from which the drilling start point in the coordinate system of the 3D data is determined, and it is assumed that this is the 3D position of the drill tip (i.e., the anchor point) at the first time point. After the tool has been advanced into the patient, a further X-ray image is acquired at a second time point. Ambiguity regarding the 3D drill pose can be resolved here based on the assumption that the drill axis determined at the second time point passes through the drilling start point determined at the first time point. This assumption requires that the drilling actually begins at the anchor point, as determined at the first time point. This assumption can be particularly easily justified when a surgical robot is used, as unintended movement of the surgical object is eliminated and patient movement can be detected by single-image registration. Patient movement may also be detected in real time by a head-mounted augmented reality device (for example, using a camera integrated into the augmented reality device). At any point, if the system detects patient movement, the system may request a new X-ray image, request a repetition of the procedure, or use the detected movement of the anatomical structure to improve the estimate of the drill pose relative to the anatomical structure.

[0100] If the patient's movement is sufficiently large compared to the expected movement of the drill bit, simply detecting the movement of the drill bit (without tracking an independent object) may be sufficient to detect that the patient has moved. This can be used to alert the surgeon, for example, by estimating that the drill may have shifted from its desired position.

[0101] Similarly, if respiratory motion is sufficiently greater than drill tip motion, it may suffice to identify respiratory motion as an element of drill tip motion and model it as a periodic motion that can be subtracted from drill tip position in order to improve the accuracy of the 3D position of the drill tip relative to the moving anatomical structure.

[0102] An alternative, and possibly more accurate, method for tracking patient movement is to track a second object to improve the accuracy of drill tracking relative to anatomical structures within the region of interest. For example, the system may directly track the patient (e.g., visible skin or tissue around the entry point of the drill bit), while an indirect example is estimating the position of a sleeve independently of the drill bit, which may be assumed to remain on the bone surface during drilling, thereby enabling the detection of drilling depth relative to the bone surface, which is primarily moving. Overall, directly or indirectly tracking the anatomical area of ​​interest reduces noise when tracking drill position relative to the area of ​​interest, improving the accuracy of surgical steps (e.g., with respect to drill guidance for surgeons or surgical robots).

[0103] It is understood that tracking of objects such as drills, puncture devices, sleeves, or patient anatomical structures may be performed in real time or at any given time. For example, tracking may not be available throughout the entire time due to obstruction, positioning of the object resulting in unacceptable ambiguity, or spontaneous interruption of the tracking process.

[0104] To utilize drill tracking, it is beneficial to link the drill to a 3D coordinate system of 3D data or a 3D coordinate system of X-ray images. This requires tracking objects visible in the X-rays outside the X-rays to estimate the spatial relationship between the tracking system and the other 3D coordinate system. This may require some degree of system synchronization so that the tracked drill can be mapped to the drill detected in a particular X-ray. Furthermore, estimating the patient's orientation and position based on prior information may suffice to generate sufficiently accurate spatial relationships (including a calibration step). Coordinate system registration may be approached in a different way than the focus of the systems disclosed herein.

[0105] This procedure may be repeated, and in some cases, X-ray images may be acquired from other imaging directions, for example, to improve accuracy.

[0106] If the system directly controls the imaging device, new images may be acquired automatically. If the system controls a robotic imaging device, the system may automatically acquire images from a desired imaging direction.

[0107] According to one embodiment, such a system may be particularly useful in spinal surgery. The following three workflows are examples of how this system may be used to perform pedicle screw drilling for spinal fusion.

[0108] An exemplary workflow for performing pedicle screw drilling according to a plan (see Figure 4). Please note that the step numbering in the workflow in Figure 4 is a combination of "1." indicating that this is the first of three workflows, followed by the actual step number used in the following description of the workflow.

[0109] 1. The surgeon and / or system perform preoperative planning in a 3D dataset (e.g., CT scan or 3D X-ray scan) by determining the target path, i.e., the planned drilling trajectory, in the 3D dataset, where the target point is the starting point of the intended drilling trajectory located on the bone surface of the pedicle, and the target endpoint is the ending point of the intended drilling trajectory.

[0110] 2. The surgeon positions the tip of the drill on the dorsal bone surface near the pedicle to be drilled (the drill tip may be positioned more precisely if the relative position of the drill to the bone is already known).

[0111] 3. If the system utilizes external real-time (or near real-time) tracking, the system may continuously track the drill bit, drilling machine, or sleeve to determine the drill position at least partially independently of X-rays.

[0112] If the tracking system is unavailable, you may proceed to step 5.

[0113] 4. The system can continuously detect a second criterion (e.g., tracking an anatomical structure or sleeve within the region of interest) and / or model the movement of the region of interest to improve the accuracy of drill bit tracking within the region of interest.

[0114] 5. The surgeon acquires an X-ray image (for example, in an approximately anterior-posterior (AP) imaging direction), preferably holding the drilling machine so that the surgeon's hand is not in the X-ray beam. If the imaging direction results in the surgeon's hand being in the X-ray beam, the system may provide instructions for a different imaging direction.

[0115] 6. The system detects the position of the drill tip (i.e., anchor point) in the X-ray image. The system estimates the region of interest in the X-ray image by calculating digitally reconstructed radiographs (DRRs) for a number of possible position estimates from the 3D dataset. The DRR that best matches the position estimate is selected. The DRR does not need to be limited to the region of interest. The system can use a weighting function to more accurately match important areas within the 2D region of interest (e.g., if the image quality is poor), and the 2D region of interest is determined by the drill tip position in the X-ray. The weighting function may also depend on the type of C-arm (flat panel or image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D dataset (preoperative or intraoperative). Based on the position estimate, the system determines a virtual projection of the target point into the X-ray image, which may be displayed as an overlay in the X-ray image.

[0116] 7. The system determines the 3D drill tip position in the coordinate system of the 3D dataset based on the detected drill tip in the X-ray image by applying the knowledge that both the drill tip and the target point are located on the bone surface. If the distance between the drill tip (i.e., anchor point) and the target point is below a threshold (e.g., 2 mm, which may depend on the type of vertebra), the system proceeds to step 8.

[0117] Otherwise, the system issues a command to reposition the drill tip to move it closer to the target point. The surgeon follows the command and returns to step 5.

[0118] 8. If the drill tip has remaining distance to the target point, a local model of the vertebral surface near the target point may be taken into consideration when determining the 3D drill tip position. The local model may be derived from a vertebral model if the vertebra to be drilled has already been classified (i.e., which vertebral level it is, e.g., L3, L4, etc.).

[0119] 9. Based on the determined 3D drill tip position in the coordinate system of the 3D dataset, the system determines the 3D position and orientation of the drill in the coordinate system of the 3D dataset.

[0120] 10. If the deviation between the 3D drill orientation and the target path is below a threshold (e.g., less than 2°), proceed to step 12. Otherwise, the system issues a command to adjust the drill angle.

[0121] 11. The surgeon may adjust the drill angle and return to step 5 (without needing to change the imaging direction), or continue / start drilling directly and proceed to step 13. While the surgeon adjusts the drill angle, the system may update information (e.g., trajectory, position, angle, etc.) in real time or near real time.

[0122] 12. The system calculates the remaining drilling depth and issues a corresponding drilling command.

[0123] 13. The surgeon follows instructions (for example, by perforating a given distance). During the perforation procedure, the system provides information on the current perforation depth in real time or near real time. Whenever confirmation of the perforation trajectory is desired or recommended by the system (for example, after perforating a certain distance or when perforating near an important anatomical structure), new X-ray images may be acquired (without changing the imaging direction).

[0124] 14. The system performs matching, detection, etc., as described above. Based on the actual drilling start point (which may be, for example, the position of the drill tip shown in the X-ray image when the first drilling command is given), the system determines the 3D position and 3D orientation of the drill.

[0125] 15. If the remaining drilling distance is below the threshold (e.g., 1 mm), proceed to step 17. If the deviation between the 3D drill orientation and the target path is below the threshold (e.g., 1°), return to step 12. Otherwise, the system issues a command to adjust the drill angle. If there is a drill that is too deep, the system may provide a warning.

[0126] 16. The surgeon follows the instructions, obtains new X-ray images, and returns to step 14.

[0127] 17. The current pedicle perforation procedure is completed.

[0128] 18. Return to step 2 to perform the next pedicle perforation.

[0129] It should be noted again that all references to "surgeon" in this workflow could refer not only to human users but also to robotic surgeons, mechanical assistance devices, or similar equipment.

[0130] In general, in addition to each of the exemplary workflows disclosed herein, the system may also provide assistance for performing skin incisions. This may be particularly useful when a surgical robot or robotic arm is employed. According to one embodiment, a surgical tool (e.g., a surgical scalpel) is aligned with a target trajectory, but at a sufficiently large distance from the target point so that the surgical tool does not yet touch the skin. The robot or robotic arm may then move the surgical tool along the target trajectory toward the target point using a soft tissue protection sleeve until it encounters some resistance (i.e., touches the skin). Resistance may be automatically detected, for example, by a pressure sensor. The incision may be made through the soft tissue protection sleeve. After the incision has been made, the surgical tool may be changed (e.g., switched to a drill while holding the soft tissue protection sleeve in place), and the surgical tool may be advanced until it encounters greater resistance (i.e., contacts the bone surface). Resistance may be automatically detected, for example, by a pressure sensor. After completing the drilling of one pedicle, the procedure may be repeated for the next pedicle. The pressure sensor may be incorporated into the powered tool or robotic arm that holds the surgical object. By recording the pressure exerted by the surgical object on the anatomical structure, the system may be able to distinguish, for example, between a drilling process and a slipping motion in which the surgical object slides over the anatomical structure.

[0131] It may be beneficial if a robot or robotic arm can automatically switch surgical tools (possibly within a soft tissue protection sleeve), advance or retract the soft tissue protection sleeve, and (possibly using a vibratory drilling mode) drill through the soft tissue protection sleeve. It may be advantageous to specifically design a soft tissue protection sleeve or adapter suitable for the robot. In the design of such a sleeve, detectability in radiographic images may also be considered. For example, it may be advantageous to fabricate the sleeve from a radiolucent material, or partially from a radiolucent material (e.g., within the area covering the tip of the drill), so that the surgical object (e.g., a drill, especially its tip) is visible in radiographic images even when covered by the sleeve. It may also be advantageous to incorporate certain features into the surgical object and / or sleeve that can be used to make the detection of the surgical object in radiographic images easier.

[0132] The sleeve may also be patient-specific if partial segmentation around the target point is performed preoperatively. The sleeve may be constructed, for example, by a 3D printer and may or may not be made of a radiolucent material. The tip of the patient-specific sleeve (the geometric shape of the opening may be a closed line in 3D space) may fit to the anatomical surface of the patient's bone around the target point, based on the precise rotational alignment of the sleeve. This may have the advantage of a very good fit and may also eliminate the need for precise navigation information, as it assists the surgeon (or robot) in finding the desired position of the sleeve tip on the planned target point with great precision. This may also ensure that the sleeve tip is not positioned in the wrong 3D location, which can occur when based on suboptimal imaging, meaning that there is surface ambiguity, meaning that two points exist on a surface with different 3D coordinates but appear in the same 2D coordinates in the X-ray image.

[0133] A similar technique may be applied to position the endoscope (the tip of the endoscope's trocar may have a specific extension) at a desired distance to the surface of the patient's bone anatomical structure, thus making it possible to use an X-ray image from a single field of view, and still allowing for precise positioning of the endoscope at a desired 3D position above the bone surface.

[0134] Another technique to address the aforementioned potential ambiguity problem may be to determine the direction of the X-ray beam through the planned or current 2D location of the target point, based on the X-ray image and the determined line of sight. After determining the 5DOF location of the region of interest in the X-ray image (i.e., after determining the transformation between the X-ray image 3D coordinate system and the dataset 3D coordinate system), this beam, which is a line in the dataset's 3D coordinate system, intersects the 3D dataset. In the case of a 3D dataset that can provide density information (e.g., CT), all points (voxels) that intersect this line indicate a density value (e.g., a Hounsfield scale). By analyzing all the density values ​​of the voxels that intersect this line (e.g., analyzing the gradient), it may be possible to determine whether there are two or more intersections of the line with the bone surface in the neighborhood (e.g., less than 10 mm) around the target point. In this case, a different imaging direction may be needed to avoid ambiguity and compute accurate 3D navigation information. This procedure may be repeated for multiple imaging directions to find an imaging direction suitable for a particular target point. During the procedure, the system may advise the surgeon or robotic surgeon to position the imaging device accordingly to prevent inaccurate navigation based on the above-mentioned topological ambiguity.

[0135] The sleeve may be X-ray transparent around its tip, but not in the portion further away from the tip (for example, 40 mm above the tip). In this case, the system can also detect the drill bit inside the sleeve, while still being able to determine the 3D position of the sleeve relative to the region of interest.

[0136] A fully radiolucent sleeve can be tracked, for example, by a real-time tracking system, based on the determination of the 3D position of the drill bit based on an X-ray image at a first time point, and the knowledge that the drill was inserted into the sleeve at that first time point. The 3D position of the sleeve relative to the region of interest can be determined at a second time point. The system may be able to do the same with sleeves that are partially radiolucent, where the opaque portion of the sleeve was not sufficiently in the X-ray image to determine its 3D position relative to the region of interest.

[0137] According to one embodiment, the use of a surgical robot may also make it possible to address the movement between individual vertebrae caused by the patient's respiration. This respiration can be modeled and detected by pressure sensors. The robot may be able to maintain a constant pressure on the vertebrae when not perforating.

[0138] This procedure may also be applied, for example, to determine the position of the chisel during wedge osteotomy, where the midpoint of the chisel is used as an anchor point.

[0139] Similar procedures are also applicable when using soft tissue protection sleeves that are not made of radiolucent material, where the anchor point (e.g., the tip of a drill) may not be visible in the X-ray image. In such cases, a virtual anchor point may be used instead of the (invisible) anchor point. This virtual anchor point may be determined based on a previously acquired X-ray image in which the anchor point was visible, and on the assumption that the anchor point has not moved between the generation of the current X-ray image and the generation of the previously acquired X-ray image. Based on the current X-ray image, the plane is defined by the axis of the sleeve and the center of the X-ray beam. The virtual anchor point may be determined by orthogonally projecting the anchor point from the previous X-ray image onto that plane. The virtual anchor point may also be determined by selecting the point closest to a contour in the plane (e.g., the intersection of the plane and the bone model, or a partial 3D segmentation of the bone surface) from the anchor point in the previous X-ray image, in order to ensure that the virtual anchor point is positioned on the bone surface. If the anchor point or anchor aspect is not visible in any X-ray, it may still be detectable, for example, if other prominent points provide sufficient information. For instance, a drill bit may have a groove that is sufficiently identifiable to project the location of a virtual anchor point.

[0140] When a sleeve is used, the sleeve itself can be considered a second surgical object that helps determine the 3D spatial relationship of the drill bit (surgical object) to the region of interest. The sleeve can be detected in X-ray images, and the fact that the drill bit is inserted into the drill sleeve (i.e., the axis of the drill sleeve constrains the axis of the drill bit) can be utilized. It may also be possible to track the drill sleeve using an extended reality device (such as head-mounted augmented reality glasses).

[0141] In the following, we will consider this an example of a “aligned” surgical tool where the axes of the drill and sleeve are constrained to each other, even if it may still be necessary to consider the possibility of bending or wobbling within the sleeve.

[0142] Another application of the procedure may be tumor resection. In this case, a preoperatively acquired MRI scan (e.g., including the tumor contour or planned resection volume) may be fused with a preoperatively or intraoperatively acquired 3D X-ray scan, and one or more target points may be identified in the MRI scan and / or 3D X-ray scan. The surgical instrument may be a resection instrument with anchor points. By determining the relative 3D position and orientation of the resection instrument and the tumor contour (or planned resection volume), accurate 3D navigation instructions may be provided even if several target points are defined in the plan. The resection may also be performed at a specific distance from the target points. Furthermore, accuracy may be improved by using a robot. The robot may receive relative 3D positioning information from, for example, an internal sensory device, so that the 3D position and orientation of the instrument relative to the resection volume are available at any point after the final determination based on the acquisition of X-ray images. Verification of this information is possible at any point by acquiring additional X-ray images, even without using anchor or target points. A virtual X-ray image is generated, assuming the imaging direction has not changed, based on the available information from the positioning sensory device, as well as the final determination of the instrument's 3D position and orientation relative to the excision volume based on the last X-ray image. After acquiring additional X-ray images from the same imaging direction, the additional images are compared to the virtual X-ray image, and the algorithm determines whether the similarity is sufficiently high (positioning information is still valid) or not (there may have been unknown movement). In the latter case, new images with determination of anchor and target points may be required.

[0143] This procedure may also be combined with a real-time navigation and tracking system. Since the procedure overlaps with existing technologies and also offers greater diversification, such a combination can result in extremely robust navigation, and this level of safety could enable autonomous robotic surgery.

[0144] An exemplary workflow for performing pedicle screw drilling (without target point planning, single image, but with knowledge of the bone surface) (see Figure 5). Similar to the previous workflow, the step numbering in the workflow in Figure 5 is a combination of "2." indicating that the workflow is the second of three, followed by the actual step number used in the following description of the workflow.

[0145] 1. The surgeon positions the tip of the drill on the dorsal bone surface, approximately in the vicinity of the pedicle to be drilled.

[0146] 2. The surgeon acquires an X-ray image (for example, roughly in the anterior-posterior (AP) imaging direction), preferably holding the drill so that the surgeon's hand is not in the X-ray beam. If the imaging direction results in the surgeon's hand being in the X-ray beam, the system may provide instructions for a different imaging direction.

[0147] 3. The system detects the position of the drill tip (i.e., anchor point) in the X-ray image. The system localizes the region of interest in the X-ray image by calculating digitally reconstructed radiographs (DRRs) for multiple imaging directions from the 3D dataset (this may have already been done before acquiring the X-ray image). The DRR that best matches the X-ray image is acquired to localize the area of ​​interest. The DRR does not need to be limited to the region of interest. The system may use a weighting function to more accurately match important areas within the 2D region of interest (e.g., if the image quality is poor), and the 2D region of interest is determined by the drill tip position in the X-ray. The weighting function may also depend on the type of C-arm (flat panel or image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D dataset (preoperative or intraoperative). The system determines the lines in the 3D dataset defined by the anchor point and the direction of the X-ray beam. The system determines a point on this line where the line intersects the bone surface (for example, by starting from the furthest point on the line, observing the voxels on this line, and determining the voxel with the highest gradient grayscale on this line, and possibly around this point). This point defines the location of the anchor point in the 3D dataset.

[0148] 4. Based on the determined 3D drill tip position in the coordinate system of the 3D dataset, the system determines the 3D position of the drill in the coordinate system of the 3D dataset. Based on the 3D dataset, the system may display the current position in different views, such as axial view and sagittal view, and may extend the drill trajectory to better visualize the corresponding drilling path. The system may also display detected anatomical structures, such as bone cortex, and may further indicate measurements of those structures, or may suggest constraints, such as cortex near buffer zones that should not be crossed.

[0149] 5. The surgeon may adjust the position and / or angle of the drill tip and return to step 2 (without needing to change the imaging direction), or directly begin / continue drilling and proceed to step 6. While the surgeon adjusts the position and / or orientation, the system updates the current position and orientation in different views (e.g., axial and sagittal) in real time or near real time.

[0150] 6. To improve accuracy, new X-ray images may be acquired (without changing the imaging direction) whenever it is desirable to confirm the drilling trajectory, or whenever the system advises otherwise (for example, when using a robot for drilling) (for example, after drilling a certain distance or when drilling near an important anatomical structure).

[0151] 7. The system performs matching, detection, etc., as described above. Based on the actual drilling start point (which may be, for example, the position of the drill tip shown in the X-ray image when the first drilling command is given), the system determines the 3D position of the drill. Based on the 3D dataset, the system displays the current position of the drill in different views, e.g., axial view and sagittal view. If the surgeon does not accept the current position as good, the surgeon may adjust the angle (e.g., while the drill bit is in motion) and proceed to step 6.

[0152] 8. The system provides information on the current drilling depth in real time or near real time. Based on this information, the surgeon determines whether the drilling procedure to the current pedicle is complete.

[0153] 9. Return to step 1 to perform the next pedicle perforation.

[0154] Furthermore, all references to "surgeon" in this workflow can refer not only to human users but also to robotic surgeons, mechanical assistance devices, or similar equipment.

[0155] Exemplary workflow for performing pedicle screw drilling (no target point planning, dual image) (see Figure 6) Similar to the previous workflow, the step numbering in the workflow in Figure 6 is a combination of "3." indicating that this is the third of three workflows, followed by the actual step number used in the following description of the workflow.

[0156] 1. If the system utilizes external real-time (or near real-time) tracking, the system may track objects continuously as they become available. Possible objects include drill bits, drilling machines, or sleeves that at least partially determine the drill position independently of X-rays. If the tracking system is unavailable, the process may proceed to step 3.

[0157] 2. Furthermore, the system may continuously detect a second criterion (e.g., tracking an anatomical structure within the region of interest or the sleeve positioned on the bone surface of an anatomical structure within the region of interest) and / or model the movement of the region of interest, in order to enable differentiation between, for example, the patient's breathing movement and the drill penetration depth, since the sleeve tip can always remain on the bone surface regardless of respiratory movement, and / or to improve the accuracy of drill bit tracking relative to the region of interest.

[0158] 3. The surgeon positions the tip of the drill on the dorsal bone surface, approximately in the vicinity of the pedicle to be drilled. The surgeon then begins drilling a short distance (e.g., 3 mm) in approximately the required direction.

[0159] 4. The surgeon acquires an X-ray image (for example, roughly in the anterior-posterior (AP) imaging direction), preferably holding the drilling machine so that the surgeon's hand is not in the X-ray beam. If the imaging direction results in the surgeon's hand being in the X-ray beam, the system may provide instructions for a different imaging direction.

[0160] 5. The system detects the position of the drill tip (i.e., anchor point) in the X-ray image. The system localizes the region of interest in the X-ray image by calculating digitally reconstructed radiographs (DRRs) for multiple imaging directions from the 3D dataset (this may have already been done before acquiring the X-ray image). The DRR that best matches the X-ray image is acquired to localize the region of interest. The DRR does not need to be limited to the region of interest. The system may use a weighting function to more accurately match important areas within the 2D region of interest (e.g., if the image quality is poor), and the 2D region of interest is determined by the drill tip position in the X-ray. The weighting function may also depend on the type of C-arm (flat panel or image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D dataset (preoperative or intraoperative).

[0161] 6. The surgeon takes further X-ray images. The angle of the drill may change between the time the image in step 4 was taken and this image, but it is necessary to ensure that the drill tip remains in place, which may be easy as the drill tip is already inside the bone (see step 3). Alternatively, the surgeon may have drilled between the first X-ray image and the further X-ray image while the drill axis of the further X-ray remained near the anchor point (where the tip was positioned) of the first X-ray.

[0162] 7. The system detects the position of the drill tip (i.e., the anchor point) in the subsequent X-ray images and estimates the location of the region of interest in the subsequent X-ray images.

[0163] 8. The system registers both X-ray images based on the location estimation of the region of interest. Based on this image registration and the detected drill tip, the system calculates the epipolar line of the detected drill tip and determines the 3D drill tip position in the coordinate system of the 3D dataset by calculating the nearest point between the epipolar lines. If the surgeon drills between the first image and the subsequent image, the epipolar lines will move further apart. However, the axis of the drill in the subsequent image represents a plane in the 3D coordinate system, and its intersection with the epipolar line of the drill tip in the first X-ray provides the 3D drill tip position in the first X-ray image. This is sufficiently accurate if the line of sight to view the drill changes sufficiently, for example, if it changes by 10 to 120 degrees between the first image and the subsequent image. If successful, proceed to step 9; otherwise, proceed to step 6.

[0164] 9. Based on the determined 3D drill tip position (from at least the first image) in the coordinate system of the 3D dataset, the system determines the 3D position and orientation of the drill in the coordinate system of the 3D dataset (in both images). Based on the 3D dataset, the system may display the current position in different views, such as an axial view and a sagittal view, and extend the drill trajectory to better visualize the corresponding drilling path.

[0165] 10. The surgeon may adjust the position and / or angle of the drill tip and return to step 4 (without needing to change the imaging direction), or directly begin / continue drilling and proceed to step 11. While the surgeon adjusts the position and / or orientation, the system updates the current position and orientation in different views (e.g., axial and sagittal) in real time or near real time.

[0166] 11. To improve accuracy, new X-ray images may be acquired (without changing the imaging direction) whenever it is desirable to confirm the drilling trajectory, or whenever the system advises otherwise (for example, when using a robot for drilling) (for example, after drilling a certain distance or when drilling near an important anatomical structure).

[0167] 12. The system performs matching, detection, etc., as described above. Based on the actual drilling start point (which may be, for example, the position of the drill tip shown in the X-ray image when the first drilling command is given), the system determines the 3D position and 3D orientation of the drill. Based on the 3D dataset, the system displays the current position of the drill in different views, e.g., axial view and sagittal view. If the surgeon does not accept the current position as good, the surgeon may adjust the angle (e.g., while the drill bit is in motion) and proceed to step 11.

[0168] 13. The system provides real-time or near-real-time information on the current drilling depth. Based on this information, the surgeon determines whether the drilling procedure to the current pedicle is complete.

[0169] 14. Return to step 3 to perform the next pedicle perforation.

[0170] Embodiments are illustrated and described in detail in the drawings and the foregoing description, but such illustrations and descriptions should be considered illustrative or exemplary rather than limiting, and the present invention is not limited to the disclosed embodiments.

[0171] Other variations of the disclosed embodiments may be understood and realized by those skilled in the art in carrying out the claimed invention, based on the study of the drawings, disclosures, and the appended claims. In the claims, the words “equipment” or “include” do not exclude other elements or steps, and the indefinite articles “a” or “an” do not exclude plurals. A single processor or other unit may perform the functions of several of the items described in the claims.

[0172] The mere fact that certain means are described in mutually different dependent claims does not mean that combinations of these means cannot be used advantageously. [Explanation of Symbols]

[0173] 1.O1 Contour 1. O2 contour 1.P1 3D position 1.P2 3D position 1. SO Surgical Objects 1.T8 vertebra T8 2.EL white line 2.P1 point 2.P2 points 2.T8 Vertebra T8 3.EL white line 3.P1 point 3.P2 points 3.T8 Vertebra T8

Claims

1. A system for determining the 3D position of a surgical object relative to a region of interest within the anatomical structure of a patient, wherein the system comprises a processing unit and a computer program product including instructions, and when the instructions are executed on the processing unit of the system, Receiving a first X-ray image generated in a first imaging direction at a first time point, wherein the X-ray image is a 2D projection image defining a first 3D image coordinate system, and the 2D projection image depicts part of the patient's anatomical structure and at least part of the surgical object, and the surgical object has a geometric aspect. Receiving a 3D dataset describing a portion of the patient's anatomical structure and the region of interest within a 3D data coordinate system, In order to determine the spatial relationship between the first 3D image coordinate system and the 3D data coordinate system, the position of a portion of the patient's anatomical structure is estimated based on the first X-ray image. The 3D position of the surgical object with respect to the region of interest in the 3D data coordinate system at the first time point is defined as follows: (I) The determined spatial relationship between the first 3D image coordinate system and the 3D data coordinate system, (II) The spatial relationship between the surgical object and the first 3D image coordinate system, and (III) Information relating to the geometric aspect They will make the decision based on that. The spatial relationship between the surgical object and the first 3D image coordinate system is, (A) Inference using neural networks, (B) Using multiple virtually generated X-ray images, wherein the images depict parts of the surgical object from various imaging directions, and (C) Applying the algorithm using a 3D model representing at least a portion of the surgical object. Derived by at least one of the groups, The information relating to the geometric aspect is (a) The geometric aspect of the surgical object is positioned near the surface of at least a portion of the region of interest. (b) Points, axes, or planes in the first 3D image coordinate system that constrain the possible options for the spatial relationship of the geometric aspect with respect to the first 3D image coordinate system. (c) A 3D position of the target point in the 3D data coordinate system that is close to the geometric aspect, wherein the projection of the target point in the 2D projection image and the projection of the geometric aspect in the 2D projection image are close to each other. The information includes at least one piece of information from the group, system.

2. The computer program product includes instructions, and when the instructions are executed on the processing unit of the system, the system Receiving a second X-ray image generated in a second imaging direction, wherein the position of the geometric aspect of the surgical object is similar in the 3D data coordinate system, and To determine the spatial relationship between the second 3D image coordinate system and the 3D data coordinate system, the position of the part of the anatomical structure is estimated based on the second X-ray image. Have them do it, Determining the 3D position of the surgical object with respect to the region of interest in the 3D data coordinate system at the first time point is: (IV) The determined spatial relationship between the second 3D image coordinate system and the 3D data coordinate system, and (V) Based on the spatial relationship between the geometric aspect of the surgical object and the second 3D image coordinate system, The spatial relationship between the geometric aspect of the surgical object and the second 3D image coordinate system is, (A) Inference using neural networks, (B) Using multiple virtually generated X-ray images, wherein the images depict parts of the surgical object from various imaging directions, and (C) Applying the algorithm using a 3D model representing at least a portion of the surgical object. Derived by at least one of the groups, The system according to claim 1.

3. The system according to claim 1 or 2, wherein the 3D dataset is updated based on at least the first X-ray image and / or the second X-ray image.

4. The system according to any one of claims 1 to 3, wherein the surgical object is a drill bit, and the region of interest within the anatomical structure of the patient is a bone structure.

5. The surgical object has a principal axis, and the first X-ray image and / or second X-ray image depict at least a portion of the surgical object together with at least a portion of a further surgical object having a principal axis and an end point, and the principal axis of the surgical object and the principal axis of the further surgical object are aligned with each other in 3D space, and the further surgical object is a drill bit. The computer program product includes instructions, and when the instructions are executed on the processing unit of the system, the system The 3D position of the further surgical object with respect to the region of interest within the 3D data coordinate system, the spatial relationship of the surgical object with respect to the region of interest, and (i) the position of the tip of the further surgical object in the first and / or second X-ray image, (ii) A physical scale in the further surgical object indicating the depth of entry of the further surgical object, and (iii) The depth of entry provided by the internal sensors of the robot holding the further surgical object. The decision is made based on at least one of the groups. The system according to any one of claims 1 to 3.

6. The surgical object has a deformation such that its tip deviates from the principal axis of the surgical object, and the computer program product includes instructions, and when the instructions are executed on the processing unit of the system, (i) the 3D position of the tip point in the 3D data coordinate system, and (ii) The geometric aspect of the surgical object in the 3D data coordinate system, such as the projected trajectory of the tip of the surgical object. To determine one or more of the group, the deformation of the surgical object is detected by fusing the updated position with the determined 3D position of the surgical object. The system according to claim 1.

7. The computer program product includes instructions, and when the instructions are executed on the processing unit of the system, the system The 3D position of the surgical object relative to the region of interest at the second time point is defined as follows: (i) the 3D position of the surgical object relative to the region of interest at the first time point, and (ii) Information relating to the change in the 3D position of the surgical object between the first time point and the second time point, wherein the information is provided by tracking based on at least one of a group of 3D sensors, planned movements of a robotic arm, a plurality of 2D cameras, and a head-mounted augmented reality device. To make a decision based on that, The system according to any one of claims 1 to 4.

8. The computer program product includes instructions, and when the instructions are executed on the processing unit of the system, the system The insertion depth of the further surgical object within the region of interest at the second time point is (i) Tracking by camera-based and / or sensor-based tracking of a tool attached to the further surgical object, (ii) Knowledge that the tip of the further surgical object was positioned on the surface of the region of interest at the first time point, (iii) Knowledge of the insertion depth of the further surgical object at the first time point, (iv) The 3D position of the further surgical object in the 3D data coordinate system at the second time point, (v) Knowledge of the distance between the tip of the further surgical object and the geometric aspect of the surgical object. The decision is made based on at least one of the groups. The system according to claim 5.

9. The computer program product includes instructions, and when the instructions are executed on the processing unit of the system, the system The 3D position of the further surgical object relative to the region of interest at the second time point is, (i) the insertion depth of the further surgical object at the second time point, and (ii) The 3D position of the surgical object relative to the portion of the region of interest at the second time point. The system according to claim 5, which causes a decision to be made based on the above.

10. The system according to claim 1, wherein the robot holds the surgical object and determines the insertion depth into the region of interest based on the robot's internal sensory data between the first and second time points.

11. The system according to any one of claims 1 to 10, further comprising a robot, the robot being configured to control the position and changes in the position of at least one of the group of imaging devices, the surgical object, further surgical objects, and the region of interest.

12. A method for determining the 3D position of a surgical object relative to a region of interest within the anatomical structure of a patient, wherein the method is Receiving a first X-ray image generated in a first imaging direction at a first time point, wherein the X-ray image is a 2D projection image defining a first 3D image coordinate system, and the 2D projection image depicts part of the patient's anatomical structure and at least part of the surgical object, and the surgical object has a geometric aspect. Receiving a 3D dataset describing a portion of the patient's anatomical structure and the region of interest within a 3D data coordinate system, To estimate the location of the region of interest based on the first X-ray image, Based on the estimated region of interest, determine the spatial relationship between the first 3D image coordinate system and the 3D data coordinate system. The 3D position of the surgical object with respect to the region of interest within the 3D data coordinate system is (I) The determined spatial relationship between the first 3D image coordinate system and the 3D data coordinate system, (II) The spatial relationship between the surgical object and the first 3D image coordinate system, and (III) Information relating to the geometric aspect This includes making decisions based on the following: The spatial relationship between the surgical object and the first 3D image coordinate system is, (A) Inference using neural networks, (B) Using multiple virtually generated X-ray images, wherein the images depict various parts of the surgical object from various imaging directions, and (C) Applying the algorithm using a 3D model representing at least a portion of the surgical object. Derived by at least one of the groups, The information relating to the geometric aspect is (a) The geometric aspect of the surgical object is positioned near the surface of at least a portion of the region of interest. (b) Points or axes in the first 3D image coordinate system that constrain the possible choices regarding the spatial relationships of the geometric aspects, (c) A 3D position of the target point in the 3D data coordinate system that is close to the geometric aspect, wherein the projection of the target point in the 2D projection image and the geometric aspect are close to each other. The information includes at least one piece of information from the group, method.

13. A computer program product that, when executed on a processing unit of the system described in claim 1, includes an instruction to cause the system to perform the method described in claim 12.

Images