System and method for mixed reality-assisted 3D navigation for musculoskeletal surgery based on X-ray images

Abstract

When executed on the system's processing unit, the system is provided with a computer program product that includes instructions to receive an X-ray image, a 3D dataset describing a region of interest within the patient's anatomical structure, and a 3D model of a surgical object. The X-ray image is a 2D projection image depicting at least a portion of the surgical object, including anchor points, as well as the region of interest. The computer program product is configured to determine the 2D position of the anchor points in the X-ray image and the imaging direction onto the region of interest based on the X-ray image. Based on this information, the 3D position of the anchor points in a 3D coordinate system defined by the 3D dataset is determined. Furthermore, the 3D position and 3D orientation of the surgical object relative to the region of interest are determined by the computer program product based on the 3D model of the surgical object, the 3D position of the anchor points in the 3D coordinate system, and some form of tracking (e.g., an extended reality device or a robotic arm).

Description

Technical Field

[0001] The present invention relates to the field of computer-assisted surgery and robot-assisted surgery. Further, the present invention relates to a system and method for providing information regarding surgical objects and anatomical structures based on CT data, or another type of 3D X-ray scan, and X-ray images. In particular, the present invention relates to a system and method for automatically determining the spatial position and orientation of a surgical object with respect to a patient. The method can be implemented as a computer program executable on a processing unit of the system.

Background Art

[0002] Spinal surgery, particularly spinal fixation, is a commonly performed surgical procedure. However, surgery on the spine also poses a significant risk to the patient. Incorrect perforation of the spine can damage the spinal cord, which can cause serious damage to the nerves or the spinal cord's covering, leading to chronic pain, permanent paralysis, incontinence, or sexual dysfunction. To reduce these risks associated with incorrect perforation, computer-assisted navigation is already used to some extent in spinal surgery. Computer assistance relates to guiding the surgeon and ensuring that the perforation is made in the correct location and that pedicle screws are properly placed, etc. This involves determining the precise relative 3D position and 3D orientation between the surgical instrument (such as a drill) and the patient (e.g., with respect to a desired perforation trajectory).

[0003] Almost all existing navigation systems require additional procedural steps and devices such as 3D cameras, trackers, fiducials, etc. In navigation spinal surgery, most current systems use optical tracking, with a dynamic reference frame attached to the spine (for patient tracking) and fiducials attached to the surgical instruments. In that case, all the fiducials must always be visible from the 3D camera. Such an approach has a number of drawbacks including, but not limited to, the following. ● The validity and accuracy of the alignment must be continuously monitored. Alignment may need to be repeated if the tracker moves. ● If the tracker's movement goes unnoticed, the navigation instructions may become incorrect, potentially harming the patient. ●Accuracy decreases as the distance from the camera increases.

[0004] In summary, existing navigation systems are prone to errors. The long chain of errors in the navigation workflow—including alignment, instrument calibration, and real-time tracking—means that the navigation system can only be as accurate as the weakest part of the chain. Therefore, even when working with a navigation system, surgeons must always ensure the accuracy of navigation instructions through uncertain verification procedures such as tactile and landmark testing.

[0005] Robot-assisted surgical systems are becoming increasingly popular due to the perceived precision they offer. However, since robots operate primarily based on commands provided by navigation systems, they can only be as accurate as the information provided by those navigation systems. Existing navigation systems are prone to errors, making current surgical robots unreliable for autonomously performing any surgical procedure. Rather, surgical robots are simply automated holding arms that hold instruments. The actual perforation still needs to be performed by the surgeon. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] The main problem with conventional navigation systems is that they do not determine the actual relative 3D position and orientation between surgical instruments and anatomical structures, but rather infer their position and orientation by using a camera to track a reference body fixed externally to the instrument (e.g., a drill) and the anatomical structure. By its nature, this camera can only see the externally fixed reference body and cannot see the drill itself inside the bone. If either of the reference bodies moves, or if the drill bends inside the bone, the navigation system may not notice this, providing false information and potentially harming the patient.

[0007] It is desirable to have a navigation system that (i) does not require any additional procedures or devices for navigation, and (ii) can reliably determine the actual 3D position and orientation of surgical objects (e.g., instruments or implants) relative to the patient. Such a navigation system may be particularly useful in spinal surgery, but can also be used in many other surgical procedures where the spatial relationship of surgical objects or implants relative to the patient must be precisely determined. Such a navigation system may be used in combination with the conventional navigation systems described above to provide redundant navigation information, and thus to make the navigation very robust, and this level of safety may enable autonomous robotic surgery. [Means for solving the problem]

[0008] At least one or the other of the above-mentioned problems is mitigated or resolved by the subject matter described in each of the independent claims. Further embodiments are described in the respective dependent claims.

[0009] A system and method are proposed that does not require a reference body or tracker to determine the 3D spatial relationship (i.e., relative 3D position and 3D orientation) of a surgical object to a portion of a patient's anatomical structure. The proposed system and method (implemented as a computer program product) is configured to determine the 3D spatial relationship based on a single X-ray image depicting at least a portion of the surgical object and at least a region of interest within the patient's anatomical structure.

[0010] Generally, when executed on the system's processing unit, a computer program product is provided that includes instructions to cause the system to receive a single X-ray image, a 3D dataset describing at least a region of interest, and a 3D model of the surgical object. The X-ray image is typically a 2D projection image depicting the region of interest within the patient's anatomical structure, as well as at least a portion of the surgical object, including anchor points. The computer program product is configured to determine the 2D position of the anchor points in the X-ray image and the imaging direction onto the region of interest based on the X-ray image. Based on this information, the 3D position of the anchor points in a 3D coordinate system defined by the 3D dataset is determined. Furthermore, the 3D position and 3D orientation of the surgical object relative to the region of interest are determined by the computer program product based on the 3D model of the surgical object and the 3D position of the anchor points in the 3D coordinate system.

[0011] The surgical object may be a surgical instrument or device such as a drill, needle, or chisel, or it may be an implant such as a pedicle screw, nail, or plate. The surgical object has a known geometric shape described by a 3D model. The surgical object also has points called anchor points, which are known in the 3D model of the surgical object and can be identified in an X-ray image. The anchor point may be, for example, the tip of a drill. In the case of a chisel, the anchor point may be on one edge or in the middle of the chisel blade.

[0012] A 3D dataset describing a portion of a patient's anatomical structure can be generated by a computed tomography (CT) scan or some other type of 3D X-ray scan, which can be obtained preoperatively or intraoperatively. The 3D dataset includes a region of interest, which must be sufficiently rigid. The region of interest may be, for example, a bone fragment, a specific vertebra, or a volume containing two adjacent vertebrae, provided that these are assumed to be sufficiently rigid. The region of interest may also be the entire 3D dataset. Since the 3D dataset accurately describes the patient's anatomical structure, at least within the region of interest, a one-to-one correspondence between points in the material world and points in the 3D dataset can be established, at least within the region of interest. Thus, the 3D dataset can define a 3D coordinate system in physical 3D space. The objective may be to determine the 3D position and orientation of surgical objects in this 3D coordinate system, in other words, to determine the 3D spatial relationship of surgical objects (in the material world) to the region of interest within the patient's anatomical structure.

[0013] According to one embodiment, target points within a 3D dataset and region of interest are determined during preoperative or intraoperative planning. These target points may be determined manually by the surgeon, automatically by the system, or semi-automatically by the surgeon with some automated assistance from the system. Thus, the location of the target points in the 3D coordinate system defined by the 3D dataset is known. The target points may be selected such that the anchor points of the surgical object can be positioned at (or near) the physical location of the target points in the patient, which can be achieved by selecting target points on the bone surface.

[0014] In another embodiment, the target point may be the starting point of the target path, and its ending point is called the target endpoint. The target path lies within the region of interest, and its position and orientation in the 3D dataset may be determined manually by the surgeon during preoperative or intraoperative planning, automatically by the system, or semi-automatically by the surgeon with some automated assistance from the system. Thus, if the target path is available, its position and orientation in the 3D coordinate system are known. For example, when treating a vertebra, the target path may correspond to a planned drilling trajectory (typically a straight line), and the target point is located on the surface of the vertebra.

[0015] The method disclosed herein teaches a method for determining the 3D position and 3D orientation of a surgical object in a 3D coordinate system defined by a 3D dataset describing the anatomical structure of a patient. In doing so, the relative 3D position and 3D orientation of the surgical object with respect to a region of interest in the patient's anatomical structure are determined. Therefore, if the region of interest includes a planned target path, the relative 3D position and 3D orientation of the surgical object with respect to the target path within the patient's anatomical structure can also be determined. It may also be intended to provide instructions for aligning the surgical object with the target path (if available), for example, for perforating along the target path.

[0016] It should be noted that some surgical objects, such as drill bits, possess rotational symmetry around their axis. In the case of such objects, rotation around the object's axis is irrelevant, so it is sufficient to determine the five degrees of freedom of their 3D spatial relationship (3D position and 3D orientation) to the patient's anatomical structure. For example, in the case of a drill bit, it is sufficient to determine the 3D position of the drill tip and the 3D orientation of the drill axis.

[0017] In the context of this invention, the rotation of a surgical object about its axis may be irrelevant to the 3D spatial relationship of the surgical object with respect to the anatomical structure. In other words, the 3D position and 3D orientation are considered determined regardless of whether the actual rotation angle of the surgical object about its axis of rotation is precisely determined or not.

[0018] Neither a target point nor a target path is necessarily required. In one embodiment, it is possible to operate using only an anchor point (e.g., the tip of a drill). In this case, the surgeon can determine during surgery whether the 3D position and orientation of the surgical object relative to the patient's anatomical structure is satisfactory, and if not, how it should be corrected.

[0019] It is emphasized that 3D datasets do not need to be segmented. In particular, there is no need to identify bone surfaces within the 3D dataset (other than selecting target points on the bone surface). Furthermore, the precise location and shape of the region of interest within the 3D dataset do not need to be known or explicitly determined. For example, when treating a vertebra, a CT scan may include this vertebra and several adjacent vertebrae (and surrounding soft tissue). Initially, the vertebra does not need to be identified within the 3D dataset. A rough approximation of the region of interest within the 3D dataset can be derived from the target points. Anchor points define the 2D region of interest in the X-ray image. For subsequent calculations, data points within the 2D region of interest may be weighted, with points further away from the anchor points receiving lower weightings.

[0020] The imaging direction on a 3D region of interest depicted in an X-ray image can be determined by calculating digitally reconstructed radiographs (DRRs) for multiple imaging directions from a 3D dataset. Alternatively, a model of the imaged structure may be used to determine the imaging direction. The imaging direction may also be determined based on position and / or motion sensors provided in the imaging device.

[0021] Throughout this disclosure, the term “imaging direction” (also called “line of sight direction”) on an object (or region of interest) means the 3D angle at which the X-ray beam passes through a selected point on the object (or region of interest). In other words, the imaging direction on a region of interest describes the orientation of the X-ray machine relative to anatomical structures within the region of interest.

[0022] When taking the DRR that best matches the X-ray image to determine the imaging direction, it may also be possible to find the optimal DRR using a numerical optimizer. The DRR does not need to be limited to the region of interest. When evaluating the best match between the DRR and the X-ray, the region of interest may be emphasized (e.g., with appropriate weighting). If the acquired X-ray image is distorted (i.e., if the X-ray image was acquired using an image intensifier rather than a flat-panel receiver), the distortion may be stronger with increasing distance between points, so it may be necessary to use a smaller area in the X-ray image for DRR matching (or emphasize points closer to the anchor point). Since distortion typically affects areas further away from the central X-ray beam more strongly, it may be useful to ensure that the anchor point is close to the center of the X-ray image.

[0023] As discussed in detail in U.S. Patent Application Publication No. 2021 / 0248779, the 3D orientation (i.e., 3D position and orientation) of thin surgical objects such as drill bits cannot always be uniquely determined based on a single X-ray image. Therefore, generally, when only one X-ray image is available, there may be ambiguity in the relative 3D spatial relationship between the surgical object and the patient's anatomical structure. This disclosure teaches how to resolve such ambiguity by first determining the 3D position of anchor points in a 3D coordinate system defined by a 3D dataset. Various methods for achieving this are disclosed.

[0024] A method according to one embodiment that does not use a target point requires that the anchor point be located on the bone surface. Based on a single X-ray image, the 2D position of the anchor point is determined within the X-ray image, and the imaging direction of the X-ray image onto the region of interest can be determined from DRR matching. In 3D space, the virtual projection onto the X-ray image determines a line by a point on the anchor point. Since the anchor point is on the bone surface, the position of the anchor point in 3D can be found by determining a point on the line on the bone surface. It is sufficient to determine the location of the bone surface along the line. Segmentation of the entire bone surface is not necessary.

[0025] Another method establishes a correspondence between an anchor point (on the surgical object) and a target point (within the 3D dataset). This is done by virtually projecting the target point into the X-ray image and utilizing, for example, the known imaging direction onto the region of interest determined by DRR matching.

[0026] If the location of the anchor point coincides with the location of the target point in the X-ray image, it can be assumed that the 3D position of the anchor point coincides with the 3D position of the target point in physical space. This is because the target point is selected such that the anchor point of the surgical instrument can be placed on the target point (within physical space). Of the three coordinates of the 3D position, two degrees of freedom are determined by the X-ray image, and the third degree of freedom is determined by the prior information that the target point and the anchor point are on or at a defined distance from the bone surface.

[0027] In other cases, for example, since perforation of the pedicle has already begun, the anchor point may not be on the bone surface. In such cases, yet another method may be applied to determine the 3D position of the anchor point in the 3D coordinate system defined by the 3D dataset. According to this method, additional X-ray images generated from different imaging directions are used. These additional X-ray images may be acquired at an earlier point in time, or (e.g., if a G-arm is used) at the first point in time simultaneously with the first X-ray image, or (e.g., if a robot capable of holding the surgical object stationary is used) at a later point in time. The additional X-ray images depict at least a portion of the surgical object, including the anchor point and the region of interest within the patient's anatomical structure. The 2D position of the anchor point in the additional X-ray image is determined. The imaging direction of the additional X-ray image onto the region of interest is determined, thereby enabling alignment of the additional X-ray image with the first X-ray image. This method assumes that there was no movement of the anchor point relative to the anatomical structure between the generation of the two X-ray images. The advantage of this method is that preoperative planning (e.g., defining the drilling trajectory and / or target point) is unnecessary. Furthermore, if the anchor point (e.g., the tip of the drill) is below the bone surface, the possibility of undesirable slippage of the surgical instrument is minimized.

[0028] Typically, a 3D dataset (e.g., a CT scan) includes scaling information (i.e., information regarding size) of the anatomical structures described in the dataset. Additionally, the scale of the scene depicted in the X-ray can also be provided by a described surgical object whose exact geometric shape is known, or another surgical object of known geometric shape that can be depicted in the X-ray image (e.g., a previously implanted pedicle screw). This redundancy regarding scale information can be used to distinguish between different surgical objects in the X-ray image that have the same geometric shape but different sizes (e.g., drills with different diameters) by comparing the scale of the 3D data with the size of the surgical object. If the acquired X-ray image is distorted, the same information (i.e., the 3D dataset, the 3D model of the surgical object, or another surgical object of known geometric shape) can also be used to correct the distortion.

[0029] If the location of an anchor point does not coincide with the location of a (virtually projected) target point in the X-ray image, the system may provide instructions on how to move the surgical instrument so that the anchor point approaches the target point, after which a new X-ray can be acquired. If the location of the anchor point is near the target point in the X-ray image, and a local model of the bone surface near the target point is available, the 3D position of the anchor point can be obtained from the local model relative to the 2D position of the anchor point detected in the X-ray image. This is sufficient to determine the 3D relative spatial relationship between the surgical object and the patient's anatomical structure. Such a local model of the bone surface may be, for example, a representative shape of the bone, a first-order approximation of the bone shape, or a local segmentation of the bone surface near the target point. The correct model may be selected based on labeling of a 3D dataset (e.g., classifying vertebrae), and the labeling may be performed automatically, semi-automatically, or manually. The size of the target point neighborhood may be fitted based on the local model; for example, the smaller the variation near the target point in the local model, the larger the neighborhood may be selected. Naturally, if segmentation of the 3D dataset (i.e., a model of the entire bone surface) is available, this can be taken into account, which may improve accuracy.

[0030] It is emphasized again that the precise location and shape of the 3D region of interest within the 3D dataset do not need to be known. Furthermore, it is not necessary to establish a precise correspondence between the 2D region of interest in the X-ray image and the 3D region of interest in the 3D dataset. However, there may be cases where several possible 3D regions of interest exist. For example, in spinal surgery, multiple vertebrae will be treated, so there may be multiple 3D regions of interest. In such cases, a human surgeon can manually identify which 3D region of interest should be treated. It may also be possible to automatically select the relevant 3D region of interest corresponding to the anchor point indicated by the surgical instrument by performing DRR matching for each possible 3D region of interest and selecting the best such match.

[0031] The region of interest is an approximation of an area (or volume) that can be assumed to be sufficiently rigid so that the 3D dataset describes the patient's anatomical structures within the region of interest with sufficient accuracy. For example, because the spine is flexible, individual vertebrae can move relative to one another. If the patient is moved after the 3D dataset has been acquired, some relative movement of individual vertebrae must be expected. This means that the area that can be assumed to be rigid may be larger from intraoperative 3D X-rays (without subsequent patient movement) than from preoperative CT scans (with subsequent patient movement), assuming no significant movement due to respiration.

[0032] The methods taught herein can also complement existing navigation techniques. Another aspect may involve the continuous incorporation of information from intraoperative radiography. The systems and methods disclosed herein do not require cameras or other sensors for navigation, but can be combined with cameras or other sensors for navigation (e.g., sensors mounted on a robot) (to improve accuracy and / or redundancy). Information from radiographic images and information from cameras or other sensors can be combined to improve the accuracy of determining relative spatial position and orientation, or to resolve any remaining ambiguities (which may result from, for example, occlusion). If the instrument is held by a robot or robotic arm, information provided by the robot or robotic arm itself (e.g., about the movements it has made) may also be considered.

[0033] The methods taught in this disclosure can also be combined with extended reality devices such as head-mounted augmented reality glasses or mixed reality glasses. For example, by tracking a power tool with a head-mounted extended reality device, changes in the 3D position and orientation of a drill bit inserted into the power tool can be determined between two points in time. Thus, if the 3D position and orientation of a surgical object (such as a drill bit) relative to an anatomical structure are known at a first point in time, the 3D position and orientation of the surgical object relative to the anatomical structure can be determined at a second point in time by tracking the power tool with an extended reality device. If it is known that only the tilt of the surgical object has changed between the two points in time (e.g., the power tool has only been tilted, and the drill tip remains in place), this knowledge can be used by the system.

[0034] Since robotic arms can typically track their movement in 3D space, it should be noted that information regarding changes in the 3D position and orientation of a surgical object between two points in time can also be provided by the robotic arm holding the surgical object.

[0035] Possible applications for applying this disclosure include, for example, drilling any type of bone for screw insertion into the pedicle, wedge osteotomy, screwing into the sacroiliac joint, screwing two vertebrae together, screwing through the pelvis, screwing through the acetabulum, or drilling for the cruciate ligaments. The teachings of the disclosure may be used, for example, for drilling, reaming, milling, chiseling, sawing, resection, and implant positioning, and thus may assist, for example, osteotomy, tumor resection, and total hip arthroplasty.

[0036] It will be understood that the system may include processing units, and the method may be implemented as a computer program product that can be executed on those processing units.

[0037] According to one embodiment, the system and method may cause an imaging device to generate an X-ray image or some type of 3D X-ray scan (e.g., a CT scan). Additionally or alternatively, the system and method may control the imaging device to move to a new position for generating an X-ray image from a different imaging direction. Such a different imaging direction may be a suitable imaging direction proposed by the system. The current imaging direction may be determined based on motion sensors and / or position sensors inside the imaging device.

[0038] It should be noted that the image data of the processed X-ray image may be received directly from the imaging device, for example, a C-arm, G-arm, or a two-plane 2D X-ray device, or it may be received from a database. A two-plane 2D X-ray device has two X-ray sources and receivers offset by any angle.

[0039] According to one embodiment, objects in an X-ray image can be automatically classified and identified within the X-ray projection image. However, objects may also be manually classified and / or identified within the X-ray projection image. Such classification or identification can be assisted by the device by automatically referencing structures recognized by the device.

[0040] It should be noted that the processing unit may be implemented by a single processor that performs all steps of the process, or by a group or multiple processors that do not need to be located in the same place. Cloud computing, for example, allows processors to be located anywhere. For example, the processing unit may be divided into a first subprocessor that controls user interaction, including a monitor for visualizing results, and a second subprocessor (possibly located elsewhere) that performs all the calculations. The first or other subprocessor may also control the movement of the C-arm or G-arm of an X-ray imaging device, for example.

[0041] According to one embodiment, the device may further include storage means that provide a database for storing, for example, X-ray images. It will be understood that such storage means may also be provided in a network to which the system may be connected, and that data related to a neural network may be received through that network. Furthermore, the device may include an imaging unit for generating at least one 2D X-ray image, the imaging unit may be capable of generating images from different directions.

[0042] According to one embodiment, the system may include a device for providing information to the user, the information including at least one of a group consisting of X-ray images and instructions relating to steps of a procedure. It will be understood that such a device may be a monitor or augmented reality device for visualization of the information, or a speaker for providing the information acoustically. The device may further include input means for manually determining or selecting the location or part of an anatomical structure in an X-ray image, such as the contour of a bone, for example, to measure distance in the image. Such input means may be a computer keyboard, computer mouse, or touchscreen for controlling a pointing device, such as a cursor on a monitor screen that may be included in the device. The device may also include a camera or scanner for reading labels on packaging or for otherwise identifying surgical objects. The camera may also allow the user to communicate visually with the device by gesture or imitation, for example, by virtually touching the device displayed in virtual reality. The device may also include a microphone and / or speaker to communicate acoustically with the user.

[0043] According to one embodiment, the system may also comprise an Extended Reality Device. The Extended Reality Device will be understood to encompass all devices on the Extended Reality spectrum and is therefore a superset of virtual reality devices, augmented reality devices, and mixed reality devices. This represents all devices that a user can wear and that (1) alter the user's perception through any sense, including sight, hearing, smell, taste, and touch, and (2) track their environment through various sensor information, including cameras, depth sensors, accelerometers, and magnetometers. These sensors may track an environment that includes imaging devices and / or at least one object, such as an anatomical model, surgical instrument, or implant. It should be noted that the tracking information allows for the distinction between objects and / or devices. For example, the Extended Reality Device may be head-mounted, through which the user sees the environment, but may also comprise glasses configured to visualize information or virtual representations of objects within the user's field of view.

[0044] It should be noted that all references to C-arm movement in this disclosure always refer to relative repositioning between the C-arm and the patient. Therefore, any translation or rotation of the C-arm can generally be replaced by the corresponding translation or rotation of the patient / operating table, or a combination of C-arm translation / rotation and patient / operating table translation / rotation. This may be particularly relevant when dealing with limbs, as it may actually be easier to move the patient's limbs than to move the C-arm. It should be noted that the required patient movement generally differs from the C-arm movement, and in particular, patient translation is usually unnecessary when the target structure is already in the desired position in the X-ray image. The system may calculate adjustments to the C-arm and / or patient. It should be further noted that all references to the C-arm may also apply to the G-arm, O-arm, etc.

[0045] The methods and techniques disclosed herein may be used in systems assisting a human user or surgeon, or in systems in which some or all of the steps are performed by a robot. Accordingly, references to “user” or “surgeon” in this patent disclosure may refer to a human user, as well as a surgical robot, mechanical assistance device, or similar apparatus. Similarly, whenever it is mentioned that instructions are given for adjusting a C-arm, it should be understood that such adjustments may be performed without human intervention, i.e., automatically, by the robotic C-arm, by the robotic operating table, or by operating room staff with some automated assistance. It should be noted that surgical robots and / or robotic C-arms may operate with greater precision than humans, potentially requiring fewer repetitions for repetitive procedures and allowing for the execution of more complex instructions (e.g., combinations of multiple repetitive steps). A key difference between robots and human surgeons is that robots can keep instruments completely still between the acquisition of two X-ray images.

[0046] The computer program product may preferably be loaded into the random access memory of a data processor. Thus, a data processor or processing unit of a system according to one embodiment may be equipped to perform at least a portion of the described process. Furthermore, this disclosure may relate to computer-readable media such as a CD-ROM on which the computer program product of the disclosure may be stored. However, the computer program product may also be presented over a network such as the World Wide Web and can be downloaded from such a network to the random access memory of a data processor. Furthermore, the computer program product may also be executed on a cloud-based processor, and the results may be presented over a network.

[0047] It should be noted that prior information regarding surgical objects (e.g., drill bit size and type) can be obtained before or during surgery by simply scanning the packaging (e.g., barcode) or by any writing on the surgical object itself.

[0048] As is evident from the above description, the primary aspect is the processing of X-ray image data that enables the automatic interpretation of visible objects. The methods described herein should be understood as methods to assist in surgical procedures on patients. Therefore, according to one embodiment, the methods do not necessarily include steps of surgical treatment of animals or humans.

[0049] It will be understood that the steps of the methods described herein, in particular the steps of the methods described in relation to the workflow of embodiments in which some are visualized in the figures, are major steps, and these major steps may be distinguished or divided into several substeps. Furthermore, there may be additional substeps between these major steps. It will also be understood that only a portion of the whole method may constitute the present invention, i.e., steps may be omitted or summarized.

[0050] It should be noted that the embodiments described refer to different subject matter. In particular, some embodiments are described by reference to method-type claims (computer program products), and other embodiments are described by reference to apparatus-type claims (systems / devices). However, those skilled in the art will understand from the above and below descriptions that, unless otherwise specified, any combination of features belonging to one type of subject matter, as well as any combination of features relating to different subject matters, is disclosed in this application.

[0051] The embodiments, as well as further embodiments, features, and advantages defined above, can also be derived from the examples of embodiments described below, and will be explained with reference to the examples of embodiments shown in the figures, but the present invention is not limited thereto. [Brief explanation of the drawing]

[0052] [Figure 1] This shows 2D X-ray projection images depicting the spine from the anterior-posterior direction. [Figure 2] This shows a slice of the same 3D scan of the spine as in Figure 1, depicting a sagittal section. [Figure 3] This shows a slice of the same 3D scan as Figure 2, depicting an axial cross-section. [Figure 4] This document illustrates an exemplary workflow for performing pedicle screw drilling based on a plan. [Figure 5] This provides a brief, exemplary workflow for performing pedicle screw drilling with knowledge of the bone surface without changing the imaging direction. [Figure 6] This document illustrates an exemplary workflow for performing pedicle screw drilling based on further X-ray images with different imaging orientations. [Modes for carrying out the invention]

[0053] Throughout the drawings, unless otherwise specified, the same reference numerals and numerals are used to indicate similar features, elements, components, or parts of the illustrated embodiments. Furthermore, while this disclosure is described in detail here with reference to the drawings, this disclosure is described in relation to exemplary embodiments and is not limited to the specific embodiments illustrated in the drawings.

[0054] This disclosure teaches a system and method for determining the 3D position and orientation of a surgical object relative to a portion of a patient's anatomical structure called a region of interest, based on a single X-ray image, which is described by a 3D dataset such as a CT scan or any other type of 3D X-ray scan. Without additional information, the 3D posture (i.e., 3D position and orientation) of a thin surgical object, such as a drill, cannot necessarily be uniquely determined based on a single X-ray image.

[0055] Figure 1 shows a 2D X-ray projection image depicting the spine from the anterior-posterior direction. A surgical object (1.SO) is positioned so that its tip lies on the bone surface of vertebra T8 (1.T8). Two contours of the surgical object are shown for two different 3D positions (1.P1 and 1.P2, respectively) and 3D orientations of the surgical object, whose contours (white solid lines labeled 1.O1 and 1.O2, respectively) are approximately identical. The tip of the surgical object in 3D space lies on the line defined by the drill tip and the focus of the X-ray machine, i.e., the epipolar line of the tip of the surgical object. Depending on the 3D position of the tip of the surgical object, the 3D orientation can be adjusted so that the projected contour of the surgical object matches its appearance in the 2D X-ray image. It is emphasized that there are countless combinations of 3D positions and corresponding 3D orientations that lead to nearly identical contours of the surgical object, two of which are depicted in Figure 1.

[0056] Figure 2 shows a slice of the same 3D scan of the spine as in Figure 1, depicting a sagittal section. The white line (2.EL) indicates all possible 3D positions of the tip of the surgical object, corresponding to the epipolar line of the tip of the surgical object from Figure 1. Two points on the line have been selected, one (2.P1) being the point where the tip of the surgical object is on the bone surface of vertebra T8 (2.T8), and the other (2.P2) being the point at a specific distance from the bone surface. Although the 3D orientation of the surgical object differs depending on the 3D position of the surgical object, it is clearly visible that both result in nearly identical contours in the X-ray projection image in Figure 1.

[0057] Figure 3 shows a slice of the same 3D scan as Figure 2, depicting an axial cross-section. The white line (3.EL) indicates all possible 3D positions of the tip of the surgical object, corresponding to line 2.EL in Figure 2. As in Figure 2, two points on the line are selected, one (3.P1) being the point where the tip of the surgical object is on the bone surface of vertebra T8 (3.T8), and the other (3.P2) being a point at a specific distance from the bone surface. Again, although the 3D orientation of the surgical object is different, it can be seen that both result in nearly identical contours in the X-ray projection image in Figure 1.

[0058] In other words, Figures 1 through 3 show two different 3D constellations with two different drill poses that yield identical (or nearly identical) X-ray projection images. The two drill constellations differ in imaging depth (the distance from the image plane, i.e., from the X-ray receiver) and inclination (i.e., 3D orientation). However, if it is possible to eliminate ambiguity regarding imaging depth (due to some prior information), the drill pose relative to the patient's anatomical structure can be uniquely determined.

[0059] This disclosure provides guidance on eliminating ambiguity regarding imaging depth by utilizing prior information that target points and anchor points are on or at a defined distance from the bone surface to establish a correspondence between the anchor points of a surgical object and target points (whose positions in a 3D coordinate system defined by a 3D dataset are known). This can be done based on a single X-ray image.

[0060] Alternatively, ambiguity regarding imaging depth may be eliminated without using a target point. According to one embodiment, the tip of the drill (i.e., the anchor point) is positioned on the bone surface near the intended drilling path (i.e., within the region of interest), and an X-ray image is acquired. The position of the drill tip is detected in the X-ray image. For an imaging direction onto the region of interest determined based on DRR matching of the 3D dataset to the X-ray image, the possible drill tip positions in 3D space on the detected anchor point, whose virtual projection onto the X-ray image, determine a line (a so-called epipolar line) in the 3D coordinate system defined by the 3D dataset. Given prior knowledge that the drill tip is on the bone surface, the position of the drill tip in the coordinate system defined by the 3D dataset can be found by determining a point on the line that intersects (i.e., is on) the bone surface. Finding the bone surface along the line can be done automatically, for example, by evaluating Hounsfield values ​​in the 3D dataset along (or in the vicinity of) the line and searching for strong gradients. Naturally, if local segmentation of the bone surface within the relevant region is already available, this may be used to intersect the lines.

[0061] By acquiring additional X-ray images from the current imaging direction and / or different imaging directions, it may be possible to improve the accuracy of determining the spatial relationship between the surgical object and the region of interest.

[0062] It is also possible to determine the 3D spatial relationship of the surgical object to the region of interest after the surgical object has been inserted into the patient, i.e., after the anchor point is no longer located on the bone surface. This is important, for example, when performing a drill. In such a case, at a first time point, an X-ray image is acquired from which the drilling start point in the coordinate system of the 3D data is determined, and it is assumed that the drilling start point is the 3D position of the drill tip (i.e., the anchor point) at the first time point. After the instrument has been advanced into the patient, at a second time point, an additional X-ray image is acquired. Here, ambiguity regarding the 3D drill pose can be resolved based on the assumption that the drill axis determined at the second time point passes through the drilling start point determined at the first time point. This assumption requires that the drilling actually started at the anchor point determined at the first time point. This assumption may be particularly easy to justify when a surgical robot is used, because this eliminates unintended movement of the surgical object and allows patient movement to be detected by single-image alignment. Patient movement may also be detected in real time by an extended reality device, such as a head-mounted augmented reality device (for example, using a camera integrated into the augmented reality device). At any point, if the system detects patient movement, the system may request a new X-ray image or request that the procedure be repeated.

[0063] This procedure may be repeated, and for example, X-ray images may be acquired from other imaging directions to improve accuracy.

[0064] If the system directly controls the imaging device, new images can be acquired automatically. If the system controls the robotic imaging device, the system can automatically acquire images from the desired imaging direction.

[0065] According to one embodiment, such a system may be particularly useful in spinal surgery. The following three workflows are examples of how this system may be used to perform pedicle screw drilling for spinal fusion.

[0066] An exemplary workflow for planned pedicle screw drilling (see Figure 4) Note that the step numbering in the workflow in Figure 4 is a combination of "1." indicating that this is the first of three workflows, followed by the actual step number used in the following description of the workflow.

[0067] 1. The surgeon and / or system perform preoperative planning in a 3D dataset (e.g., CT scan or 3D X-ray scan) by determining the target path, i.e., the planned drilling trajectory, in the 3D dataset, where the target point is the starting point of the intended drilling trajectory located on the bone surface of the pedicle, and the target endpoint is the endpoint of the intended drilling trajectory.

[0068] 2. The surgeon positions the tip of the drill on the dorsal bone surface near the pedicle to be drilled (the drill tip can be positioned more precisely if the relative position of the drill to the bone is known).

[0069] 3. The surgeon acquires an X-ray image (for example, in the approximately anterior-posterior (AP) imaging direction), preferably holding the drilling machine so that the surgeon's hand is not in the X-ray beam. If the imaging direction results in the surgeon's hand being in the X-ray beam, the system may provide instructions for a different imaging direction.

[0070] 4. The system detects the position of the drill tip (i.e., anchor point) in the X-ray image. The system determines the imaging direction onto the region of interest in the X-ray image by calculating digitally reconstructed radiographs (DRRs) for a number of imaging directions from the 3D dataset. To determine the imaging direction, the DRR that best matches the X-ray image is taken. The DRR does not need to be limited to the region of interest. The system may use a weighting function (e.g., if the image quality is poor) to more accurately match important areas within the 2D region of interest, which is determined by the position of the X-ray drill tip. The weighting function may also depend on the C-arm type (flat panel vs. image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D dataset (preoperative vs. intraoperative). Based on the imaging direction, the system determines a virtual projection of the target point into the X-ray image, which may be displayed as an overlay within the X-ray image.

[0071] 5. The system determines the 3D drill tip position in the coordinate system of the 3D dataset based on the detected drill tip in the X-ray image by applying the knowledge that both the drill tip and the target point are on the bone surface. If the distance between the drill tip (i.e., anchor point) and the target point is below a threshold (e.g., 2 mm, the threshold may depend on the type of vertebra), the system proceeds to step 6.

[0072] Otherwise, the system instructs the surgeon to reposition the drill tip to move it closer to the target point. The surgeon follows the instructions and returns to step 3.

[0073] 6. If the drill tip has remaining distance to the target point, the determination of the 3D drill tip position may take into account a local model of the vertebral surface near the target point. The local model can be derived from a vertebral model if the vertebra to be drilled has already been classified (i.e., which vertebral level, e.g., L3, L4, etc.).

[0074] 7. Based on the determined 3D drill tip position in the coordinate system of the 3D dataset, the system determines the 3D position and orientation of the drill in the coordinate system of the 3D dataset.

[0075] 8. If the deviation between the 8.3D drill orientation and the target path is below a threshold (e.g., less than 2°), proceed to step 10. Otherwise, the system instructs the system to adjust the drill angle.

[0076] 9. The surgeon adjusts the drill angle and returns to step 3 (without needing to change the imaging direction), or continues / starts the drilling directly and proceeds to step 11. While the surgeon adjusts the drill angle, the system may update information (e.g., trajectory, position, angle, etc.) in real time or near real time.

[0077] 10. The system calculates the (remaining) drilling depth and provides the corresponding drilling instructions.

[0078] 11. The surgeon follows instructions (e.g., perforating to a given distance). During the perforation procedure, the system provides information on the current perforation depth in real time or near real time. Whenever the system requests or advises confirmation of the perforation trajectory (e.g., after perforating to a certain distance or when perforating near an important anatomical structure), new X-ray images can be acquired (without needing to change the imaging direction).

[0079] 12. The system performs matching, detection, etc., as described above. Based on the actual drilling start point (which may be, for example, the position of the drill tip shown in the X-ray image when the initial drilling instruction is given), the system determines the 3D position and 3D orientation of the drill.

[0080] 13. If the remaining drilling distance is below a threshold (e.g., 1 mm), proceed to step 15. If the deviation between the 3D drill orientation and the target path is below a threshold (e.g., less than 1°), return to step 10. Otherwise, the system will instruct you to adjust the drill angle. If there is a drill that is too deep, the system may provide a corresponding warning.

[0081] 14. The surgeon follows instructions, obtains new X-ray images, and returns to step 12.

[0082] 15. The drilling procedure for the current pedicle is completed.

[0083] 16. Return to step 2 to perform the next pedicle perforation.

[0084] It should be noted again that all references to "surgeon" in this workflow refer to human users, but may also refer to surgical robots, mechanical assistance devices, or similar equipment.

[0085] In general, in addition to each of the exemplary workflows disclosed herein, the system may also provide assistance for making skin incisions. This may be particularly useful when a surgical robot or robotic arm is used. According to one embodiment, a surgical instrument (e.g., a scalpel) is aligned with a target trajectory, but at a sufficiently large distance from the target point so that the instrument does not yet touch the skin. The robot or robotic arm may then move the surgical instrument along the target trajectory toward the target point using a soft tissue protection sleeve until it encounters some resistance (i.e., contact with the skin). Resistance may be automatically detected, for example, by a pressure sensor. The incision may be made through the soft tissue protection sleeve. After the incision has been made, the surgical instrument may be changed (e.g., switched to a drill while holding the soft tissue protection sleeve in place), and the surgical instrument may be advanced until it encounters greater resistance (i.e., contact with the bone surface). Resistance may be automatically detected, for example, by a pressure sensor. After completing the drilling of one pedicle, the procedure may be repeated for the next pedicle. Pressure sensors can be incorporated into powered instruments or robotic arms that hold surgical objects. By recording the pressure exerted by the surgical object on anatomical structures, the system may be able to distinguish, for example, a drilling process from a sliding motion in which the surgical object slides over an anatomical structure such as a bone surface.

[0086] It could be useful if a robot or robotic arm could automatically switch surgical instruments (perhaps within a soft tissue protection sleeve), advance or retract the soft tissue protection sleeve, and perforate through the soft tissue protection sleeve (perhaps using a vibratory perforation mode). It may be advantageous to specially design a soft tissue protection sleeve or adapter suitable for the robot. The design of such a sleeve may also take into account detectability in radiographic images. For example, it may be advantageous to fabricate the sleeve from a radiolucent material, or partially from a radiolucent material (e.g., in the area covering the tip of a drill), so that the surgical object (e.g., a drill, especially its tip) is visible in radiographic images even when covered by the sleeve. It may also be advantageous to incorporate certain features into the surgical object and / or sleeve that can be used to make detection of the surgical object in radiographic images easier.

[0087] According to one embodiment, the use of a surgical robot may also make it possible to address the movement between individual vertebrae caused by the patient's respiration. This respiration can be modeled and detected by pressure sensors. The robot may be able to maintain constant pressure on the vertebrae when not perforating.

[0088] This procedure may also be applied, for example, to determine the position of the chisel during a wedge osteotomy in which the midpoint of the chisel is used as an anchor point.

[0089] A similar procedure may also be applicable when using soft tissue protection sleeves that are not made of radiolucent material, where anchor points (e.g., drill tips) may not be visible in the X-ray image. In such cases, virtual anchor points may be used instead of (invisible) anchor points. These virtual anchor points may be determined based on previously acquired X-ray images in which the anchor points were visible, and under the assumption that there has been no movement of the anchor points between the generation of the current X-ray image and the generation of the previously acquired X-ray image. Based on the current X-ray image, a plane is defined by the axis of the sleeve and the center of the X-ray beam. The virtual anchor points may be determined by orthogonally projecting the anchor points from the previous X-ray image onto that plane. The virtual anchor points may also be determined by selecting the point closest to the anchor points from the previous X-ray image to a contour in the plane (e.g., the intersection of the plane and a bone model, or partial 3D segmentation of the bone surface) to ensure that the virtual anchor points are positioned on the bone surface.

[0090] When a sleeve is used, the sleeve itself can be considered a second surgical object that helps determine the 3D spatial relationship of the drill bit (surgical object) to the region of interest. The sleeve can be detected in X-ray images, and it can be utilized that the drill bit is inserted into the drill sleeve (i.e., the axis of the drill sleeve is on the axis of the drill bit). It may also be possible to track the drill sleeve using an extended reality device (such as head-mounted augmented reality glasses).

[0091] Another application of the procedure may be tumor resection. In this case, a preoperatively acquired MRI scan (e.g., including the tumor contour or planned resection volume) may be fused with a preoperatively or intraoperatively acquired 3D X-ray scan, and one or more target points may be identified in the MRI scan and / or 3D X-ray scan. The surgical instrument may be a resection instrument with anchor points. By determining the relative 3D position and 3D orientation of the resection instrument and the tumor contour (or planned resection volume), precise 3D navigation instructions may be provided even if several target points are defined in the plan. Resection may also be performed at a specific distance from the target points. The use of a robot may lead to improved accuracy. Since the robot may receive relative 3D positioning information from, for example, an internal sensory device, the 3D position and 3D orientation of the instrument relative to the resection volume may be available at any point after the final determination based on the acquisition of X-ray images. Verification of this information may be possible at any point by acquiring additional X-ray images, even without using anchor points or target points. In other words, a virtual X-ray image is generated based on the available information from the positioning sensor and the final determination of the instrument's 3D position and orientation relative to the excised volume based on the last X-ray image, assuming that the imaging direction has not changed. After acquiring additional X-ray images from the same imaging direction, the additional images are compared to the virtual X-ray image, and an algorithm determines whether the similarity is sufficiently high (positioning information is still valid) or not (there may have been unknown movement). In the latter case, a new image with the determination of anchor points and target points may be required.

[0092] This procedure may also be combined with a real-time navigation and tracking system. Since the procedure is redundant to existing technologies and offers an even higher level of diversification, such a combination makes navigation highly robust, and this level of safety may enable autonomous robotic surgery.

[0093] An exemplary workflow for pedicle screw drilling (see Figure 5) (without target point planning, single image, but with knowledge of the bone surface). Similar to the previous workflow, the step numbering in the workflow in Figure 5 is a combination of "2." indicating that the exemplary workflow is the second of three, followed by the actual step number used in the following description of the workflow.

[0094] 1. The surgeon positions the tip of the drill on the dorsal bone surface, approximately in the vicinity of the pedicle to be drilled.

[0095] 2. The surgeon acquires an X-ray image (for example, in the approximately anterior-posterior (AP) imaging direction), preferably holding the drilling machine so that the surgeon's hand is not in the X-ray beam. If the imaging direction results in the surgeon's hand being in the X-ray beam, the system may provide instructions for a different imaging direction.

[0096] 3. The system detects the position of the drill tip (i.e., anchor point) in the X-ray image. The system determines the imaging direction onto the region of interest in the X-ray image by calculating digitally reconstructed radiographs (DRRs) for a number of imaging directions from the 3D dataset (this may have already been done before acquiring the X-ray image). To determine the imaging direction, the DRR that best matches the X-ray image is taken. The DRR does not need to be limited to the region of interest. The system may use a weighting function (e.g., if the image quality is poor) to more accurately match important areas within the 2D region of interest, which is determined by the position of the X-ray drill tip. The weighting function may also depend on the C-arm type (flat panel vs. image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D dataset (preoperative vs. intraoperative). The system determines the lines in the 3D dataset defined by the anchor point and the center of the beam. The system determines a point on this line where it intersects the bone surface (for example, by starting from the outermost point of the line, observing the voxels along this line, and determining the voxel on this line that has the highest grayscale gradient, possibly around this point). This point defines the location of an anchor point in the 3D dataset.

[0097] 4. Based on the determined 3D drill tip position in the coordinate system of the 3D dataset, the system determines the 3D position and orientation of the drill in the coordinate system of the 3D dataset. Based on the 3D dataset, the system displays the current position in different views, such as axial view and sagittal view, and may extend the drill trajectory to better visualize the corresponding drilling path.

[0098] 5. The surgeon may adjust the drill tip position and / or angle, return to step 2 (without needing to change the imaging direction), or directly begin / continue drilling and proceed to step 6. While the surgeon adjusts the position and / or orientation, the system updates the current position and orientation in different views (e.g., axial and sagittal) in real time or near time.

[0099] 6. To improve accuracy, whenever the system requires or advises (for example, after drilling a specific distance or when drilling near an important anatomical structure) to verify the drilling trajectory, a new X-ray image may be acquired (without needing to change the imaging direction).

[0100] 7. The system performs matching, detection, etc., as described above. Based on the actual drilling start point (which may be, for example, the position of the drill tip shown in the X-ray image when the initial drilling instruction is given), the system determines the 3D position and 3D orientation of the drill. Based on the 3D dataset, the system displays the current position of the drill in different views, such as an axial view and a sagittal view. If the surgeon does not accept the current position as correct, the surgeon may correct and adjust the angle (for example, while the drill bit is moving) and proceed to step 6.

[0101] 8. The system provides information on the current drilling depth in real time or near real time. Based on this information, the surgeon determines whether the drilling procedure for the current pedicle is complete.

[0102] 9. Return to step 1 to perform the next pedicle perforation.

[0103] An exemplary workflow for pedicle screw drilling (see Figure 6) (without target point planning, dual image). Similar to the previous workflow, the step numbering in the workflow in Figure 6 is a combination of "3." indicating that the exemplary workflow is the third of three, followed by the actual step number used in the following description of the workflow.

[0104] 1. The surgeon positions the tip of the drill on the dorsal bone surface, approximately in the vicinity of the pedicle to be drilled. The surgeon then begins drilling a short distance (e.g., 3 mm) in approximately the required direction.

[0105] 2. The surgeon acquires an X-ray image (for example, in the approximately anterior-posterior (AP) imaging direction), preferably holding the drilling machine so that the surgeon's hand is not in the X-ray beam. If the imaging direction results in the surgeon's hand being in the X-ray beam, the system may provide instructions for a different imaging direction.

[0106] 3. The system detects the position of the drill tip (i.e., anchor point) in the X-ray image. The system determines the imaging direction onto the region of interest in the X-ray image by calculating digitally reconstructed radiographs (DRRs) for a number of imaging directions from the 3D dataset (this may have already been done before acquiring the X-ray image). To determine the imaging direction, the DRR that best matches the X-ray image is taken. The DRR does not need to be limited to the region of interest. The system may use a weighting function (e.g., if the image quality is poor) to more accurately match important areas within the 2D region of interest, which is determined by the position of the X-ray drill tip. The weighting function may also depend on the C-arm type (flat panel vs. image intensifier), the type of vertebra (cervical / thoracic / lumbar), and / or the type of 3D dataset (preoperative vs. intraoperative).

[0107] 4. The surgeon takes further X-ray images. The angle of the drill may change between taking the images from step 2 and taking these images, but it is necessary to ensure that the drill tip remains in place, which may be easy as the drill tip is already inside the bone (see step 1).

[0108] 5. The system detects the position of the drill tip (i.e., the anchor point) in the subsequent X-ray images and determines the imaging direction over the region of interest in the subsequent X-ray images.

[0109] 6. The system aligns both X-ray images based on the determined imaging direction. Based on this image alignment and the detected drill tip, the system calculates the epipolar rays of the detected drill tip and determines the 3D drill tip position in the coordinate system of the 3D dataset by calculating the nearest point between the epipolar rays.

[0110] 7. Based on the determined 3D drill tip position in the coordinate system of the 3D dataset, the system determines the 3D position and orientation of the drill in the coordinate system of the 3D dataset. Based on the 3D dataset, the system displays the current position in different views, such as axial view and sagittal view, and may extend the drill trajectory to better visualize the corresponding drilling path.

[0111] 8. The surgeon may adjust the drill tip position and / or angle, return to step 2 (without needing to change the imaging direction), or directly begin / continue drilling and proceed to step 9. While the surgeon adjusts the position and / or orientation, the system updates the current position and orientation in different views (e.g., axial and sagittal) in real time or near real time.

[0112] 9. To improve accuracy, whenever the system requires or advises (for example, after drilling a specific distance or when drilling near an important anatomical structure) to verify the drilling trajectory, a new X-ray image may be acquired (without needing to change the imaging direction).

[0113] 10. The system performs matching, detection, etc., as described above. Based on the actual drilling start point (which may be, for example, the position of the drill tip shown in the X-ray image when the initial drilling instruction is given), the system determines the 3D position and 3D orientation of the drill. Based on the 3D dataset, the system displays the current position of the drill in different views, e.g., axial view and sagittal view. If the surgeon does not accept the current position as correct, the surgeon may correct and adjust the angle (for example, while the drill bit is moving) and proceed to step 9.

[0114] 11. The system provides information on the current drilling depth in real time or near time. Based on this information, the surgeon determines whether the drilling procedure for the current pedicle is complete.

[0115] 12. Return to step 1 to perform the next pedicle perforation.

[0116] Examples of methods are provided below, and these methods are implemented as computer program products.

[0117] 1. When executed on the system's processing unit, the system will Determine the 3D position and 3D orientation of object 1 relative to object 2 at the first point in time. Based on the information received from the real-time tracking system, the 3D position and 3D orientation of object 1 relative to object 2 at a second point in time are determined. The X-ray imaging device receives a 2D X-ray image generated at a second time point, and the 2D X-ray image depicts at least a portion of object 1 and at least a portion of object 2. Receive 3D data containing object 1 or a 3D model of object 1. Receive 3D data including object 2 or a 3D model of object 2. (i) 3D data including object 1 or a 3D model of object 1, (ii) 3D data including object 2 or a 3D model of object 2, (iii) The 3D position and orientation of object 1 relative to object 2 at a second point in time, based on information received from the real-time tracking system. A virtual projection image is generated based on this, Compare 2D X-ray images with virtual projection images. Examples of computer program products that include instructions.

[0118] 2. The 3D position and 3D orientation of object 1 relative to object 2 at a second point in time, based on information received from a real-time tracking system, are verified based on a comparison of a 2D X-ray image and a virtual projection image, as in the computer program product of Example 1.

[0119] 3. The comparison is based on the computer program product in Example 2, using similarity thresholds based on different weightings for different 2D regions of the image.

[0120] 4.2 The imaging direction of the DX line image is, The process involves generating multiple virtual projection images of object 1 and selecting the virtual projection image that best fits object 1 within a 2D X-ray image. Assuming that the imaging direction of the X-ray imaging device is the same as the imaging direction determined at the previous point in time, Tracking the movement of X-ray imaging devices using a real-time tracking system. A computer program product, one of Examples 1 to 3, determined based on at least one of the following groups.

[0121] 5. A computer program product, any one of Examples 1 to 4, wherein the determination of the 3D position and 3D orientation of object 1 relative to object 2 at a first time point is based on at least one of the group consisting of acquiring multiple surface points, acquiring multiple 2D X-ray images by a tracked X-ray imaging device, acquiring an intraoperative 3D scan by a tracked imaging device, and acquiring at least one 2D X-ray image including an object having an anchor point.

[0122] 6. A computer program product of any one of Examples 1 to 5, wherein the real-time tracking system is at least one of the group consisting of a robot positioning and / or motion sensing device, a navigation system having an optical tracker, a navigation system having an infrared tracker, a navigation system having EM tracking, a navigation system utilizing a 2D camera, a navigation system utilizing Lidar, a navigation system utilizing a 3D camera, and a navigation system including wearable tracking elements such as augmented reality glasses.

[0123] 7. The 3D position and orientation of object 1 relative to object 2 at a second time point are further verified based on another determination of the 3D position and orientation of object 1 relative to object 2 at a second time point, the other determination being based on a single 2D X-ray image taking into account a specific point in one of the objects, as in the computer program product of Example 2.

[0124] A system comprising a processing unit configured to run a computer program product according to any one of the above examples may further comprise a real-time navigation and tracking system configured to provide the 3D position and 3D orientation of a first object relative to a second object. The system may further comprise a robotic device for holding and moving one of the objects. The system may further comprise an X-ray imaging device.

[0125] A method for verifying the 3D position and 3D orientation of object 1 relative to object 2 is: A step of determining the 3D position and 3D orientation of object 1 relative to object 2 at a first point in time, The steps include determining the 3D position and 3D orientation of object 1 relative to object 2 at a second point in time based on information received from a real-time tracking system, A step of receiving a 2D X-ray image generated at a second time point by an X-ray imaging device, wherein the 2D X-ray image depicts at least a portion of object 1 and at least a portion of object 2. The steps include receiving 3D data or a 3D model of object 1, The steps include receiving 3D data or a 3D model of object 2, (i) 3D data including object 1 or a 3D model of object 1, (ii) 3D data including object 2 or a 3D model of object 2, (iii) The 3D position and 3D orientation of Object 1 relative to Object 2 at the second point in time. A step of generating a virtual projection image based on, The steps involve comparing a 2D X-ray image with a virtual projection image. It may include.

[0126] Embodiments are illustrated and described in detail in the drawings and the foregoing description, but such illustrations and descriptions should be considered illustrative or exemplary rather than limiting, and the present invention is not limited to the embodiments disclosed.

[0127] Other variations of the embodiments of the disclosure may be understood and brought about by those skilled in the art in carrying out the claimed invention, based on the study of the drawings, this disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude plural. A single processor or other unit may perform the functions of multiple items described in the claims.

[0128] The mere fact that certain means are described in different dependent claims does not mean that a combination of these means cannot be used advantageously.

Claims

1. When executed on the system's processing unit, the system will The X-ray image generated at a first time point is received, and the X-ray image is a 2D projection image depicting at least a portion of the surgical object, including anchor points and regions of interest within the patient's anatomical structure. The region of interest is described, and a 3D dataset defining the 3D coordinate system is received. The 3D model of the surgical object is received, and the position of the anchor point is known in the 3D model. Determine the 2D position of the anchor point in the aforementioned X-ray image. Based on the aforementioned X-ray image, the imaging direction onto the region of interest is determined. i. The 2D position of the anchor point in the X-ray image, and ii. The imaging direction on the region of interest Based on this, the 3D position of the anchor point in the 3D coordinate system at the first time point is determined. i. The 3D model of the surgical object, and ii. The 3D position of the anchor point in the 3D coordinate system at the first time point. Based on this, the 3D position and 3D orientation of the surgical object relative to the region of interest at the first time point are determined. i. The 3D position and 3D orientation of the surgical object with respect to the region of interest at the first time point, and ii. Information relating to the changes in the 3D position and 3D orientation of the surgical object between the first time point and the second time point. Based on this, the 3D position and 3D orientation of the surgical object relative to the region of interest at the second time point are determined. Computer program products that include instructions.

2. The computer program product according to claim 1, wherein the information relating to the changes in the 3D position and 3D orientation of the surgical object is provided by at least one of the group of extended reality devices and robotic arms.

3. When the aforementioned computer program product is executed on the system's processing unit, it will be executed on the system, X-ray images are received from different imaging directions, and the X-ray images are 2D projection images depicting at least a portion of the surgical object, including the anchor point and the region of interest within the patient's anatomical structure. The 2D position of the anchor point in the further X-ray image is determined. Based on the aforementioned further X-ray images, the different imaging directions on the region of interest are determined. The further X-ray image is aligned with the X-ray image generated at the first time point. The command further includes, The determination of the 3D position of the anchor point in the 3D coordinate system at the first time point is as follows: i. The 2D position of the anchor point in the further X-ray image, ii. The different imaging directions on the region of interest, iii. Alignment of the further X-ray image with the X-ray image generated at the first time point, and iv. The assumption that the anchor point has not moved relative to the anatomical structure between the generation of the further X-ray image and the first time point. Based on A computer program product according to either claim 1 or 2.

4. The 3D dataset includes target points whose positions in the 3D coordinate system are known. The computer program product includes further instructions causing the system to determine the virtual projection of the target point onto the X-ray image generated at the first time point, The determination of the 3D position of the anchor point in the 3D coordinate system at the first time point is further based on the 2D position of the virtual projection of the target point relative to the 2D position of the anchor point in the X-ray image generated at the first time point. The computer program product according to claim 1.

5. The computer program product according to claim 4, wherein the 3D dataset is locally segmented in the vicinity of the target point.

6. The computer program product according to any one of claims 4 and 5, wherein knowledge of the position of the aforementioned target point is obtained automatically.

7. When the aforementioned computer program product is executed on the system's processing unit, it will be executed on the system, The line in the 3D coordinate system is determined, and the line's virtual projection onto the X-ray image includes a point on the anchor point in the X-ray image. To determine the point on the line on the bone surface within the region of interest. The command further includes, The computer program product according to claim 1, wherein the determination of the 3D position of the anchor point in the 3D coordinate system at the first time point is further based on the point of the line on the bone surface.

8. The computer program product according to claim 7, wherein the bone surface within the region of interest is obtained by local segmentation of the 3D dataset.

9. The aforementioned computer program product is installed on the system. A new X-ray image generated at the second time step is received, the new X-ray image being a 2D projection image depicting at least a portion of the surgical object and the region of interest. Including further orders, The information relating to the changes in the 3D position and 3D orientation of the surgical object between the first time point and the second time point is, i. The 3D position of the anchor point in the 3D coordinate system at the second time point, determined based on the new X-ray image, and ii. The assumption that the axis of the surgical object at the second time point passes through the 3D position of the anchor point in the 3D coordinate system at the first time point. A computer program product according to claim 1, including the above.

10. The aforementioned computer program product is installed on the system. A new X-ray image generated at the second time step is received, the new X-ray image being a 2D projection image depicting at least a portion of the surgical object and at least a portion of the region of interest. Determine the imaging direction of the new X-ray image. Based on the determined imaging direction of the new X-ray image, a virtual projection image is generated that includes the surgical object with the determined 3D position and 3D orientation of the surgical object relative to the region of interest at the second time point. To verify the 3D position and 3D orientation of the surgical object relative to the region of interest at the second time point, the virtual projection image is compared with the new X-ray image. The computer program product according to claim 1, including further instructions.

11. The surgical object is a drill bit, and the X-ray image further depicts the drill sleeve. The determination of the 3D position and 3D orientation of the drill bit with respect to the region of interest at the second time point is as follows: Real-time tracking of the drill sleeve using an extended reality device or robotic arm, Detection of the drill sleeve in the X-ray image, and The knowledge that the drill bit is inserted into the drill sleeve. Further based on at least one of the groups consisting of The computer program product according to claim 1.

12. The aforementioned surgical object is a drill bit. The determination of the 3D position and 3D orientation of the drill bit with respect to the region of interest at the second time point is further based on input from a pressure sensor that enables the system to distinguish between drilling into the region of interest and sliding along the region of interest. The computer program product according to claim 1.

13. The aforementioned computer program product is installed on the system. To provide the user with instructions on how to adjust the 3D position and / or 3D orientation of the surgical object. Adjusting the 3D position and / or orientation of the surgical object, To provide the user with instructions on how to adjust the imaging direction for generating X-ray images. Adjusting the imaging direction for generating X-ray images, To generate X-ray images, To detect the movement of the anatomical structures of the aforementioned patient, Providing the user with instructions to generate an X-ray image. The computer program product according to claim 1, further comprising instructions for performing at least one of the group consisting of the following.

14. A system comprising a processing unit configured to execute a computer program product according to any one of claims 1 to 13.

15. The system according to claim 14, further comprising an extended reality device.

Images