Methods, computing devices, systems, and computer program products for assisting in positioning of an instrument relative to a particular body part of a patient - Patents.com

Abstract

A computer-implemented method, computing device, system, and computer program product for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200), comprising: receiving intraoperative imaging data (ID) including 2D images of the specific body part (202) from a plurality of viewpoints and a 2D image of at least a portion of the instrument (5) from at least one viewpoint; reconstructing an anatomical 3D shape (AS) of the specific body part (202) based on the intraoperative imaging data (ID) and data indicating the viewpoints of the 2D images using an artificial intelligence based algorithm corresponding to the specific body part (202); estimating a current position (5c) of the instrument (5) relative to the anatomical 3D shape (AS) based on the intraoperative imaging data (ID); and generating positioning guidance data (GD) including the estimated current position (5c) of the instrument (5) relative to the anatomical 3D shape (AS) of the specific body part (202).

Description

[Technical field]

[0001] The present invention relates to a computer implemented method of assisting in the positioning of an instrument, such as a surgical instrument, relative to a particular body part of a patient. The present invention further relates to a computing device configured to assist in the positioning of an instrument, such as a surgical instrument, relative to a particular body part of a patient. The present invention further relates to a system for assisting in the positioning of an instrument relative to a particular body part of a patient. The present invention further relates to a computer program product comprising instructions which, when executed by a processing unit of a computing device, cause the computing device to assist in the positioning of an instrument, such as a surgical instrument, relative to a particular body part of a patient. [Background technology]

[0002] Traditional prior art computer-aided surgery and surgical navigation techniques have enabled surgeons to obtain a "glass-like" patient, where critical information from preoperative imaging and preoperative planning is directly available to the surgeon's perception. Traditional computer-aided surgical guidance systems rely on three main aspects: 1) preoperative planning based on an anatomical 3D model derived from preoperative images (such as CT or MRI scans), 2) registration of preoperative data to intraoperative anatomy, and 3) real-time tracking of surgical instruments.

[0003] Preoperative plans used in prior art surgical guidance systems show a step-by-step surgical procedure on a generated 3D anatomical model. However, these preoperative plans are usually idealized overviews of the intraoperative reality, which may be influenced by preoperative events that affect the patient's respective body part and / or intraoperative conditions such as bleeding, complications, surgical inaccuracies, etc. Thus, surgeons are often forced to revert to traditional (non-navigation) techniques.

[0004] Registration of the preoperative plan to the intraoperative anatomy is often considered another shortcoming of conventional computer-aided surgical guidance systems. In the context of this application, the term registration refers to the process of calibration / alignment of the preoperative plan to the actual real-life tissues and physical position, orientation of the patient at the time of surgery. Conventional computer-aided surgical guidance systems address this by using techniques such as matching mutual landmarks and / or features in the preoperative plan with a 3D anatomical model using optically tracked pointers, or by using image-based registration methods (i.e., 2D-3D) that automatically match preoperative images (such as CT or MRI scans) with images acquired intraoperatively. However, such common techniques in computer-aided surgery and surgical guidance systems are susceptible to various error sources and technical complications ranging from marker movement and small capture range to slow calculations.

[0005] US Patent Application Publication No. 2022 / 044440(A1) and WO2020 / 108806A1 describe a method for artificial intelligence assisted surgery using statistical shape modeling, in which objects identified from intraoperative imaging data (using an artificial intelligence algorithm) are classified in an X-ray projection image, and a 3D representation and localization of the classified objects is obtained by deforming a statistical shape model to fit the imaging of the classified objects in the X-ray image. However, statistical shape modeling has the disadvantage that individual imaged features in the intraoperative image that were not captured by the statistical shape model are lost in the process of deforming the statistical shape model to fit the intraoperative image. Thus, statistical shape modeling can provide at best a general statistical approximation of 3D dimensional shapes based on intraoperative imaging, but is not suitable for 3D reconstruction of anatomical shapes.

[0006] In a third aspect of conventional computer-assisted surgical navigation, the tracking of the three-dimensional pose of the instrument, particularly the desired surgical hardware, should be estimated within the same reference frame of the tissue of a particular body part of the patient. For this purpose, known computer-assisted surgery and surgical guidance systems rely on fiducial markers attached to the surgical instrument. However, line-of-sight problems are pointed out as a significant burden for the clinical use of such systems. Methods exist for 2D x-ray-based surgical instrument pose estimation using conventional (usually intensity-based) registration techniques. However, such techniques are prone to the same limitations as image-based registration methods.

[0007] Recently, with the introduction of fluoroscopy machines capable of intraoperative cone-beam computed tomography (CBCT), it is possible to obtain 3D volumetric images of the patient during surgery and use this data together with an optical tracking system to realize registration-free surgical navigation. In such a method, the patient is registered to the preoperative plan through the monitoring of optically tracked fiducial markers attached to the patient, which are visible both in intraoperative cone-beam computed tomography and in the optical tracking system. For the acquisition of full-range cone-beam computed tomography images, such a method results in the addition of ionizing radiation and may also result in metal artifact problems. Moreover, the main technical limitation of cone-beam computed tomography-based intraoperative navigation is the assumption of fixed fiducial markers attached to the patient tissue, which has proven to be insufficient in operating room conditions.

[0008] Although such methods have been proven to provide superior implantation accuracy compared to standard freehand surgical methods, they have not yet been widely adopted in modern operating rooms around the world. As reported in a global survey, only 11% of spinal surgeries are performed using computer-assisted navigation systems, with the majority of surgeries being performed using traditional freehand open techniques, where the surgeon relies on his or her visual and tactile feedback to place the spinal implant within the pedicle region. This is because the aforementioned computer-assisted surgical navigation methods require an extensive registration process to transfer the preoperative plan to the tissue and / or generally require the installation of external navigation hardware in the operating room. This may interfere with existing surgical workflows and add to surgical time, radiation exposure, and costs. For example, a navigation system that requires the acquisition of a cone-beam computed tomography scan during surgery can increase surgical time by up to 8.2 minutes, resulting in ionizing radiation of 2.09 to 4.81 mSV.

[0009] In summary, known computer-assisted surgery and surgical guidance systems are prone to gaze and / or marker movement problems, require extensive registration of the preoperative plan, require the installation of external navigation hardware in the operating room, and / or significantly disrupt surgical workflow. Summary of the Invention [Problem to be solved by the invention]

[0010] It is an object of the present invention to provide a method, computing device, system, and computer program product for assisting in positioning an instrument relative to a particular body part of a patient that overcomes one or more of the shortcomings of the prior art.

[0011] In particular, it is an object of the present invention to provide a registration-free method for assisting in positioning of an instrument relative to a particular body part of a patient that can reconstruct anatomical 3D shape and generate a visual representation of the position of the instrument relative to the particular body part of the patient using only intraoperative imaging data, i.e., without the need for navigation hardware to be installed in the operating room. [Means for solving the problem]

[0012] According to the present disclosure, this object is addressed by the features of independent claim 1. Further advantageous embodiments also emerge from the dependent claims and the description.

[0013] In particular, this object is achieved by a computer-implemented method for assisting in the positioning of an instrument, such as a surgical instrument (e.g., a surgical drill, knife, or a surgical laser instrument) or a medical diagnostic instrument, relative to a particular body part of a patient, comprising: receiving intraoperative imaging data; reconstructing an anatomical 3D shape using intraoperative imaging data and using artificial intelligence based algorithms corresponding to a particular body part; estimating a current position of the instrument based on the intraoperative imaging data; generating positioning guidance data including a visual representation of an estimated current position of the instrument relative to the anatomical 3D shape of the particular body part; It includes.

[0014] In certain embodiments, the steps of receiving intraoperative imaging data, estimating the current position of the instrument, and generating guidance data are performed repeatedly or continuously over a period of time in preparation for / prior to surgical treatment of the patient.

[0015] <Receiving intraoperative imaging data>

[0016] Intraoperative imaging data is received by a computing device from an imaging device disposed near a patient. In this specification, the placement of the imaging device near a patient refers to a placement that allows the imaging device to capture intraoperative imaging data for the patient. The imaging data includes a plurality of 2D images. Two or more of the 2D images are images of a particular body part of the patient from two or more different viewpoints relative to the particular body part of the patient. One or more of the same plurality of 2D images of the particular body part also are images of at least a portion of an instrument from at least one viewpoint. In the context of the present invention, the term viewpoint, with respect to the viewpoint of the imaging data, refers to the position and / or orientation (e.g., roll, pitch, yaw) of the imaging device relative to the particular body part and relative to the instrument, respectively (e.g., in an x, y, and z Cartesian coordinate system).

[0017] According to an embodiment of the present invention, the intraoperative imaging data includes one or more of: a) radiation-based images, in particular X-ray images; b) ultrasound images; c) arthroscopic images; d) optical images; and / or e) any other cross-sectional images.

[0018] Respective images of specific body parts of the patient and parts of the instrument are captured using an imaging device communicatively connected to the computing device. In the case of radiation-based images, the imaging device may include a C-arm imaging device based on X-ray technology. The C-arm imaging device comprises a generator (X-ray source) and an image intensifier or flat panel detector. A C-shaped connecting element allows for movement horizontally, vertically, and / or around a swivel axis, so that 2D X-ray images of the patient can be generated from various viewpoints around the patient. The generator emits X-rays that penetrate the patient's body. The image intensifier or detector converts the X-rays into a visible image, which is transmitted to the computing device.

[0019] The intraoperative imaging data includes data indicative of viewpoints corresponding to the multiple 2D images that identify a position and / or orientation of an imaging device capturing the multiple 2D images, such as a position in an x, y, and z Cartesian coordinate system of the imaging device relative to a particular body part, and / or an orientation as roll, pitch, and yaw. According to embodiments disclosed herein, the data indicative of viewpoints corresponding to the multiple 2D images is stored in a data store included in or communicatively connected to the computing device. Alternatively or additionally, the viewpoints corresponding to the intraoperative imaging data are estimated by the computing device based on the intraoperative imaging data.

[0020] In a particular embodiment, estimating a viewpoint corresponding to the intraoperative imaging data is performed using an instrument geometric model that describes the geometry of the instrument. First, multiple projections of the instrument geometric model are calculated from multiple candidate viewpoints. The candidate viewpoints are selected as distinct viewpoints in a predefined space of possible viewpoints of the imaging device. In other words, unrealistic positions and orientations of the imaging device relative to the patient are not considered to conserve computing power. Then, a viewpoint corresponding to the 2D image of the intraoperative imaging data is identified by comparing at least a portion of the instrument imaged in each 2D image of the intraoperative imaging data with multiple projections calculated from multiple candidate viewpoints. In particular, the comparison includes applying a matching function to identify a best match between one of the calculated projections of the "virtual" instrument (based on the instrument geometric model) from the multiple candidate viewpoints and the part of the "physical" instrument imaged by the imaging device. The candidate viewpoint that produces the best match is selected as the estimated viewpoint.

[0021] According to the embodiment disclosed herein, estimating the viewpoint corresponding to intraoperative imaging data is performed using an artificial intelligence-based algorithm trained using multiple imaging data sets with known viewpoints. To overcome the availability and / or accuracy limitations of imaging data sets with known viewpoints, multiple imaging data sets including 2D images from known viewpoints are generated from 3D imaging data, particularly computed tomography CT scans. Using this artificial intelligence-based algorithm trained before surgery, the intraoperative position of the imaging device can be estimated based only on intraoperative images without the need for external tracking devices or calibration phantoms.

[0022] <Generating anatomical 3D shapes>

[0023] The anatomical 3D shape of the particular body part is reconstructed by the computing device based on the intraoperative imaging data and data indicating viewpoints corresponding to the multiple 2D images using an artificial intelligence-based algorithm corresponding to the particular body part.

[0024] According to the embodiments disclosed herein, the anatomical 3D shape is reconstructed as a voxelized volume and / or mesh. It is important to emphasize that the artificial intelligence based algorithm must be a model corresponding to a particular body part, allowing the reconstruction of the 3D anatomical shape from multiple 2D images of the particular body part.

[0025] According to certain embodiments disclosed herein, an artificial intelligence based algorithm is trained using a number of annotated imaging datasets that image body parts (of a person other than the patient) that correspond to a particular body part of the patient. The annotations of the imaging datasets include data that identify and / or describe characteristics of the body parts, such as identifying pixels, vectors, contours, surfaces in a 2D image that image the particular body part, and / or voxels in a 3D image that image the particular body part.

[0026] To overcome the limitations of availability and / or accuracy of annotated imaging data sets of body parts corresponding to specific body parts of a patient, according to embodiments disclosed herein, multiple annotated imaging data sets are generated from annotated 3D imaging data, particularly computed tomography CT scans, of body parts corresponding to specific body parts of a patient. In particular, given an input preoperative CT scan, synthetic 2D images such as fluoroscopy shots (i.e., DRRs) are generated from various viewpoints around the patient. For example, using this method, hundreds of annotated "synthetic" 2D images (of specific body parts) can be generated from a single annotated CT scan, which are annotated "synthetic" 2D images that can be used by artificial intelligence-based algorithms to improve the ability to reconstruct accurate anatomical 3D shapes from as few 2D intraoperative images as possible.

[0027] According to an embodiment, reconstructing the anatomical 3D shape is performed in two stages: segmenting the intraoperative imaging data to identify specific body parts of the patient, and further using the segmented intraoperative imaging data to reconstruct the anatomical 3D shape.

[0028] To segment the intraoperative imaging data, a region of interest that includes a particular body part of a patient is first identified in the intraoperative imaging data using an artificial intelligence-based detection and segmentation model, such as a convolutional neural network-based detection and segmentation model. The region of interest is then semantically segmented using the artificial intelligence-based detection and segmentation model, thereby generating segmented intraoperative imaging data.

[0029] <Estimation of the current location of the device>

[0030] Once the anatomical 3D shape is reconstructed, the current position of the instrument relative to the anatomical 3D shape of the particular body part is estimated based on the intraoperative imaging data, in particular the 2D images of the imaging data in which the instrument is imaged. According to the embodiments disclosed herein, estimating the current position of the instrument is performed using an instrument geometric model that describes the geometry of the instrument. First, a projection of the instrument geometric model is compared with at least a portion of the instrument imaged in the respective 2D images of the intraoperative imaging data. The instrument geometric model is projected onto one or more planes of the 2D images of the intraoperative imaging data in which at least a portion of the instrument is imaged. The planes of the 2D images of the intraoperative imaging data are determined based on the viewpoints of the respective 2D images. Then, a position of the instrument geometric model is determined that generates a projection onto the plane of the 2D images of the intraoperative imaging data that (best) matches at least a portion of the instrument imaged in the respective 2D images of the intraoperative imaging data. In other words, a reverse process is applied compared to the (initial) determination of the viewpoint of the intraoperative imaging data. However, this inverse process does not necessarily apply to the same 2D images (of the intraoperative imaging data) that are used to determine the viewpoint of the images used for reconstruction of the anatomical 3D shape.

[0031] According to the embodiments disclosed herein, at the initial stage of the method to assist in instrument positioning, the anatomical 3D shape is reconstructed once, but estimation of the instrument's current position is performed repeatedly at set intervals and / or triggered by certain events and / or manually triggered.

[0032] To improve the accuracy of estimating the position of the instrument and / or to improve the accuracy of estimating the viewpoint corresponding to the intraoperative imaging data, according to a further embodiment, the method of the present invention further comprises providing the instrument according to an instrument geometric model. The instrument geometric model is specifically designed to optimize the estimation of the position of the instrument based on as few intraoperative images as possible. In particular, to enable the estimation of the orientation of the instrument based on 2D images, the instrument is designed such that at least a part of it is not perfectly rotationally symmetric about any of the axes of the Cartesian coordinate system. Alternatively or additionally, the instrument is designed with a special marker to facilitate the identification of the instrument based on the 2D intraoperative images.

[0033] <Generation of positioning guidance data>

[0034] Upon reconstructing the anatomical 3D shape of the particular body part and estimating the current position of the instrument, positioning guidance data is reconstructed by the computing device, including a visual representation of the estimated current position of the instrument relative to the anatomical 3D shape of the particular body part. According to embodiments disclosed herein, the guidance data is reconstructed as a 2D image displayed on a computer display. Alternatively or additionally, the guidance data is reconstructed as an augmented reality overlay that allows an augmented reality device, such as a headset, to project an overlay onto a user's field of view, whereby the overlay is aligned with the user's viewpoint of the particular body part of the patient and / or includes overlay metadata aligned with the user's viewpoint of the instrument. According to embodiments disclosed herein, a visual representation of the estimated current position of the instrument is superimposed on the visual representation of the reconstructed anatomical 3D shape.

[0035] According to embodiments disclosed herein, the computing device controls a display device to display at least a portion of the guidance data, the display device being a computer screen, an augmented reality headset, or any device configured to display the guidance data.

[0036] To guide the surgeon to correctly place the instrument, according to further embodiments disclosed herein, a predetermined position of the instrument relative to the anatomical 3D shape of the particular body part is identified by the computing device, and a visual representation of the predetermined position of the instrument is superimposed on a visual representation of the estimated current position of the instrument. The predetermined position of the instrument is retrieved or received by the computing device from a data store contained in or communicatively connected to the computing device. Alternatively or additionally, the predetermined position of the instrument is calculated by the computing device, and the predetermined position of the instrument is determined by an optimization function based on the anatomical 3D shape of the body part as well as data indicative of the surgical procedure.

[0037] Advantageously, the embodiments disclosed herein enable automatic pre-operative planning based on the reconstructed anatomical 3D shape to guide the surgeon in the placement of surgical instruments. Given that intra-operative imaging data is used to reconstruct the anatomical 3D shape of a body part (e.g., the spine), and a predetermined position / trajectory of the instrument can be identified based on the anatomical 3D shape of the body part, no pre-operative planning stage is required to define a safe implantation trajectory, and no registration of the pre-operative data to the intra-operative patient position is required.

[0038] Another object of the present invention is to provide a computing device for positioning of an instrument relative to a particular body part of a patient that can reconstruct anatomical 3D geometry and generate a visual representation of the position of the instrument relative to the particular body part of the patient using only intraoperative imaging data, i.e., without the need for navigation hardware to be installed in the operating room and without the need for a preoperative planning registration process.

[0039] According to the present disclosure, this object is addressed by the features of independent claim 15. Further advantageous embodiments emerge from the dependent claims and the description.

[0040] In particular, the above identified object is further achieved by a computing device comprising a data input interface, a data output interface, a processing device, and a storage unit. The data input interface, such as a wired (e.g., Ethernet, DVI, HDMI, VGA) and / or wireless data communication interface (e.g., 4G, 5G, Wifi, Bluetooth, UWB), is communicatively connected to the imaging device and configured to receive intraoperative imaging data from the imaging device. The data output interface, such as a wired (e.g., Ethernet, DVI, HDMI, VGA) and / or wireless data communication interface (e.g., 4G, 5G, Wifi, Bluetooth, UWB), is configured to transmit at least a portion of the guidance data to a display device communicatively connectable to the data output interface. The storage unit includes instructions, when implemented by the processing device, that cause the computing device to implement a method for assisting in positioning of an instrument according to any one of the embodiments disclosed herein.

[0041] According to an embodiment, the computing device is a stand-alone computer communicatively connected to the imaging device. Alternatively or additionally, the computing device is a remote computer (e.g., a cloud-based computer) communicatively connected to the imaging device using a communication network, in particular at least in part using a mobile communication network. Alternatively or additionally, the computing device is integrated into the imaging device or the display device.

[0042] Another object of the present invention is to provide a system for positioning of an instrument relative to a particular body part of a patient that can reconstruct anatomical 3D geometry and generate a visual representation of the position of the instrument relative to the particular body part of the patient using only intraoperative imaging data, i.e., without the need for navigation hardware to be installed in the operating room and without the need for a preoperative planning registration process.

[0043] According to the present disclosure, this object is addressed by the features of independent claim 16. Further advantageous embodiments emerge from the dependent claims and the description.

[0044] In particular, the above identified object is further achieved by a system comprising a computing device, an imaging device, and a display device according to any one of the embodiments disclosed herein, the system being configured to perform a method according to any one of the embodiments disclosed herein. The imaging device is communicatively connected to the computing device and is positioned in the vicinity of the patient to enable the imaging device to capture intraoperative imaging data of the patient, whereby two or more of the 2D images are images of a particular body part of the patient from two or more different viewpoints relative to the particular body part of the patient. One or more of the same multiple 2D images of the particular body part are also images of at least a part of the instrument from at least one viewpoint. In the case of radiation-based images as intraoperative images, the imaging device includes a C-arm imaging device based on X-ray technology. The C-arm imaging device comprises a generator (X-ray source) and an image intensifier or flat panel detector. A C-shaped connection element allows for movement in a horizontal direction, a vertical direction, and / or around a swivel axis, so that 2D X-ray images of the patient can be generated from various viewpoints around the patient. The generator emits X-rays that penetrate the patient's body. The image intensifier or detector converts the x-rays into a visible image, which is transmitted to a computing device, the display device being a computer screen, an augmented reality headset, or any device configured to display the guided data.

[0045] Another object of the present invention is to provide a computer program product for positioning of an instrument relative to a particular body part of a patient that can reconstruct anatomical 3D geometry and generate a visual representation of the position of the instrument relative to the particular body part of the patient using only intraoperative imaging data, i.e., without the need for navigation hardware to be installed in the operating room and without the need for a preoperative planning registration process.

[0046] According to the present disclosure, this object is addressed by the features of independent claim 18. Further advantageous embodiments emerge from the dependent claims and the description.

[0047] In particular, the above identified objects are addressed by a computer program product comprising instructions which, when executed by a processing unit of a computing device, cause the computing device to perform a method according to any one of the embodiments disclosed herein.

[0048] According to an embodiment, the instructions (included in the computer program product) include an artificial intelligence based algorithm that corresponds to a particular body part of the patient, the artificial intelligence based algorithm being trained using a number of annotated imaging data sets that image the body part that corresponds to the particular body part of the patient, and the annotations include data that identify and / or describe characteristics of the body part.

[0049] According to an embodiment, the instructions (included in the computer program product) include instructions for controlling an imaging device to capture intraoperative imaging data including 2D images, the plurality of 2D images capturing a particular body part of a patient from a plurality of different viewpoints of the particular body part of the patient, and one or more of the plurality of 2D images capturing at least a portion of an instrument from at least one viewpoint.

[0050] According to an embodiment, the instructions (included in the computer program product) include instructions for controlling a display device to display at least a portion of the guidance data including a visual representation of an estimated current position of the instrument, a visual representation of a reconstructed anatomical 3D shape, and / or a visual representation of a predetermined position of the instrument, etc.

[0051] It should be understood that both the general description above and the detailed description below present embodiments and are intended to provide an overview or framework for understanding the nature and character of the present disclosure. The accompanying drawings are included to provide a further understanding, and are incorporated in and constitute a part of this specification. The drawings illustrate various embodiments, and together with the description, serve to explain the principles and operation of the disclosed concepts.

[0052] The term "particular" is used herein to refer to embodiments of the invention without any indication of preference and without any indication that the feature introduced as particular is essential to all embodiments of the invention.

[0053] The invention described herein will be more fully understood from the detailed description given hereinafter and the accompanying drawings, which should not be construed as limitations on the invention described in the appended claims. [Brief description of the drawings]

[0054] [Figure 1] FIG. 1 is a highly schematic perspective view of a system for assisting in instrument positioning installed in an operating room, according to one embodiment of the present invention; [Diagram 2] 4 is a flow chart illustrating steps of a method for assisting in instrument positioning, according to one embodiment of the present invention. [Diagram 3] 1 is a flow chart illustrating steps for reconstructing an anatomical 3D shape based on intraoperative imaging data and data indicating viewpoints corresponding to a plurality of 2D images, according to an embodiment of the present invention. [Figure 4]FIG. 1 is a schematic diagram of segmenting intraoperative imaging data to identify specific body parts of a patient. [Diagram 5] FIG. 13 is a schematic diagram of a further embodiment of segmenting intraoperative imaging data, including identifying a region of interest, followed by semantic segmenting of the region of interest to identify a particular anatomy of a patient within the region of interest. [Figure 6] FIG. 13 is a schematic diagram of anatomical 3D shape reconstruction based on segmented intraoperative imaging data using artificial intelligence-based algorithms corresponding to specific body parts. [Figure 7] 13 is a flow chart showing steps of a further embodiment of reconstructing an anatomical 3D shape in multiple stages. [Figure 8] FIG. 1 is a schematic diagram of determining a predetermined position of an instrument. [Figure 9] FIG. 9A is an exemplary example of a visual representation of an estimated current position of an instrument and a visual representation of a predetermined position of an instrument superimposed on a visual representation of a reconstructed anatomical 3D shape, FIG. 9B is an exemplary example of a visual representation of an estimated current position of an instrument superimposed on a 2D image of intraoperative imaging data ID, and FIG. 9C is an exemplary example of a visual representation of an estimated current position of an instrument, a visual representation of a predetermined position of an instrument, and a visual representation of an ideal screw trajectory of a surgical implant on a visual representation of a reconstructed anatomical 3D shape. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings, in which some, but not all, features are shown. Indeed, the embodiments disclosed herein may be embodied in many different forms and are not to be construed as limitations to the embodiments set forth herein, but rather, such embodiments are provided so that this disclosure will satisfy applicable legal requirements. Wherever possible, like reference numbers are used to refer to like components or parts.

[0056] FIG. 1 shows a highly schematic perspective view of a system 1 for assisting in the positioning of an instrument 5, installed in an operating room, with a patient 200 lying on an operating table 2. As shown, the system 1 comprises a computing device 10, an imaging device 20, and a display device 30. The system 1 is shown based on an embodiment utilizing radiation-based images as intraoperative images. Thus, the imaging device 20 comprises a C-arm imaging device 20 that uses X-ray technology to capture intraoperative images. The C-arm imaging device 20 comprises a generator (X-ray source) 22. A C-shaped connecting element (C-arm) 24 allows for movement in the horizontal direction, the vertical direction, and / or around a swivel axis, so that 2D X-ray images of the patient 200 can be generated from various viewpoints around the patient. The generator 22 emits X-rays that penetrate the patient's body 200. A detector 26 converts the X-rays into imaging data ID, which is transmitted to the computing device 10.

[0057] The imaging device 20 is communicatively connected to the computing device 10 and positioned in proximity to the patient 200 enabling the imaging device 20 to capture intraoperative imaging data ID of the patient 200, whereby two or more of the 2D images are of a particular body part 202 of the patient 200 from two or more different viewpoints of the particular body part 202 of the patient 200. One or more of the same multiple 2D images of the particular body part 202 also capture at least a portion of the instrument 5 from at least one viewpoint.

[0058] In the illustrated embodiment, the display device 30 comprises a series of computer screens 32 communicatively connected to the computing device 10 and configured to display the guidance data GD.

[0059] Referring now to the flow chart of FIG. 2, the steps of a computer-implemented method for assisting in positioning an instrument 5 relative to a particular body part 202 of a patient 200 will be described.

[0060] As shown in FIG. 2, the method includes: Step S10: Capture intraoperative imaging data; Step S20: Receive intraoperative imaging data; Step S30: Reconstructing an anatomical 3D shape using intraoperative imaging data and using an artificial intelligence based algorithm corresponding to a particular body part 202; Step S40: Estimate the current position of the instrument based on the intraoperative imaging data ID; Step S50: Identifying a predetermined position 5p of the instrument; Step S60: generating positioning guidance data GD comprising a visual representation of the estimated current position 5c of the instrument 5 relative to the anatomical 3D shape AS of a particular body part 202; Step S70: outputting the positioning guidance data GD using the display device 30; This includes the following major steps:

[0061] Steps that are specific to particular embodiments are indicated in the figures with dashed lines.

[0062] In step S10, intraoperative imaging data ID is captured by an imaging device 20 positioned near a patient 200. The intraoperative imaging data ID includes data indicative of a viewpoint corresponding to the multiple 2D images, which identifies a position and / or orientation of the imaging device 20 capturing the multiple 2D images, such as a position in an x, y, and z Cartesian coordinate system of the imaging device 20 relative to a particular body part 202 of the patient 200, and / or an orientation as roll, pitch, yaw.

[0063] According to a first embodiment, data indicative of viewpoints corresponding to a plurality of 2D images are stored in a data store included in or communicatively connected to the computing device 10. The viewpoints stored in the data store are determined by tracking the C-arm 24 to estimate the imaging parameters at the time of exposure, through which the intraoperatively acquired 2D images can be assigned to respective intrinsic and extrinsic imaging parameters that substantially define the viewpoints at which the 2D images were generated. Optionally, tracking the C-arm 24 is preceded by a calibration process, a pre-operative calibration process (i.e., pre-calibration), in which the C-arm 24 is manipulated in a specific manner to cover a zigzag-shaped range of motion. During this pre-calibration phase, mathematical relationships between the tracking observations and the imaging parameters are established at specific posture intervals, which are later used to derive an interpolation function that can generate intraoperative imaging parameters based on the tracking data.

[0064] Alternatively or additionally, a viewpoint corresponding to the intraoperative imaging data ID is estimated by the computing device 10 based on the intraoperative imaging data ID. In one embodiment, a calibration algorithm extracts the viewpoint of the 2D image by placing a precisely fabricated calibration object (i.e., a phantom) containing distinct features (e.g., radiopaque features) with known geometry within the imaging field of view, and estimates the imaging parameters based on the projection of such features.

[0065] Alternatively or additionally, estimating the viewpoint corresponding to the intraoperative imaging data ID is performed using an artificial intelligence-based algorithm trained using a number of imaging data sets with known viewpoints. To overcome the availability and / or accuracy limitations of imaging data sets with known viewpoints, a number of imaging data sets including 2D images from known viewpoints are generated from 3D imaging data, in particular computed tomography CT scans. For example, simulated intraoperative fluoroscopy shots (i.e., digitally reconstructed radiographs, DRRs) generated based on preoperative CT scans with corresponding posture parameters are used to train a convolutional neural network (CNN) for regression tasks. Using this artificial intelligence-based algorithm trained before surgery, the intraoperative position of the imaging device 20 can be estimated based only on intraoperative images without the need for an external tracking device or calibration phantom.

[0066] In a subsequent step S20, the intraoperative imaging data ID is received by the computing device 10 via its data input interface 14 from the imaging device 20.

[0067] In a subsequent step S30, an anatomical 3D shape AS of the particular body part 202 is reconstructed by the computing device 10 using an artificial intelligence based algorithm corresponding to the particular body part 202 based on the intraoperative imaging data ID and data indicating viewpoints corresponding to the plurality of 2D images. A detailed description of the step S30 of reconstructing the anatomical 3D shape AS is given with reference to Figures 3, 4, 5, 6 and 7.

[0068] In step S40, the current position 5c of the instrument 5 relative to the anatomical 3D shape AS of the particular body part 202 is estimated based on the 2D images of the imaging data ID in which the instrument 5 was imaged. The current position 5c of the instrument 5 is performed based on the prior knowledge of the geometry of the instrument 5, described by an instrument geometric model. First, the projection of the instrument geometric model is compared with at least a part of the instrument 5 imaged in the respective 2D images of the intraoperative imaging data ID. The instrument geometric model is projected onto one or more planes of the 2D images of the intraoperative imaging data ID in which at least a part of the instrument 5 was imaged. The planes of the 2D images of the intraoperative imaging data ID are determined based on the viewpoint of each 2D image. Then, the position of the instrument geometric model is determined that generates a projection onto the plane of the 2D images of the intraoperative imaging data ID that (best) matches at least a part of the instrument 5 imaged in the respective 2D images of the intraoperative imaging data ID.

[0069] According to the embodiments disclosed herein, at an initial stage of the method to assist in instrument positioning, the anatomical 3D shape is reconstructed once, but estimation of the current position 5c of the instrument 5 is performed repeatedly at set intervals and / or triggered by certain events and / or manually triggered.

[0070] To improve the accuracy of estimating the position of the instrument 5, the instrument 5 is certified according to an instrument geometric model. The instrument geometric model is specifically designed to optimize the estimation of the instrument's position based on as few intraoperative 2D images as possible. In particular, to enable the estimation of the instrument's orientation based on the 2D images, the instrument is designed such that at least a part of it is not perfectly rotationally symmetric about any of the axes of the Cartesian coordinate system. Alternatively or additionally, the instrument is designed with special markers to facilitate the identification of the instrument based on the 2D intraoperative images.

[0071] In step S50, a predetermined position 5p of the instrument 5 relative to the anatomical 3D shape AS of a particular body part 202 is identified by the computing device 10, and a visual representation of the predetermined position 5p of the instrument 5 is overlaid on a visual representation of the estimated current position 5c of the instrument 5 to assist the surgeon in correctly placing the instrument 5.

[0072] One embodiment for determining the predetermined position 5p of the instrument 5 will now be described with reference to FIG.

[0073] FIG. 3 shows a flow chart illustrating steps of reconstructing an anatomical 3D shape AS based on intraoperative imaging data ID and data showing viewpoints corresponding to a plurality of 2D images. As shown, reconstructing an anatomical 3D shape AS is performed in two stages: step S32-segmenting the intraoperative imaging data ID to identify a specific body part 202 of the patient 200, and step S34-further using the segmented intraoperative imaging data ID to reconstruct an anatomical 3D shape AS. Step S32-segmenting the intraoperative imaging data ID to identify a specific body part 202 of the patient 200 using an artificial intelligence-based detection and segmentation model is shown in FIG. 4, which is applied to segmenting an intraoperative 2D image of the spine to identify individual vertebrae. To train the artificial intelligence-based detection and segmentation model, synthetic X-rays (i.e., DRRs) are generated from various viewpoints around the patient 200 given an input preoperative CT scan. The CT scans used for this purpose may be collected through a public dataset including CT scans and corresponding vertebral level annotations. For example, using this method, a training database of over 40,000 annotated 2D images can be created from as few as 200 pre-operative CT scans.

[0074] FIG. 5 shows a schematic diagram of a further embodiment of step S32 of segmenting the intraoperative imaging data ID by a two-stage approach, including identification of a region of interest and semantically segmenting the region of interest to identify a specific body part 202 of the patient 200 within the region of interest. To segment the intraoperative imaging data ID, a region of interest including a specific body part 202 of the patient 200 is first identified in the intraoperative imaging data ID using an artificial intelligence-based detection and segmentation model, such as a convolutional neural network-based detection and segmentation model. The region of interest is then semantically segmented using the artificial intelligence-based detection and segmentation model, thereby generating a segmented intraoperative imaging data ID. Supervised learning is used to train the artificial intelligence-based detection and segmentation model used for segmentation in step S32. First, a convolutional neural network CNN-based detection model is trained to identify individual body parts (vertebral levels in the illustrated example) on the 2D image by detecting the coordinates of bounding boxes each including a single body part (a single vertebra). The identified bounding boxes are then used as regions of interest for cropping the intraoperative 2D images. Furthermore, an end-to-end segmentation model is trained to semantically segment the projections of the vertebrae within the regions of interest. During the inference phase, the intraoperative x-ray images are fed into the segmentation model, which generates a semantic segmentation for each vertebral level (used for 3D reconstruction).

[0075] FIG. 6 illustrates a method for segmenting intraoperative imaging data ID using an artificial intelligence based algorithm corresponding to a particular body part 202 and a viewpoint P corresponding to multiple 2D images. 1-n 1 shows a schematic diagram of a reconstruction of an anatomical 3D shape AS of a particular body part 202 based on data indicative of the anatomical 3D shape of the vertebrae in the 2D image. As shown, the segmented intraoperative imaging data ID is back-projected into a 3D coordinate system to create an anatomical 3D shape for each body part 202 (vertebrae). The back-projection is performed from the viewpoint P of the 2D image.1-n , where each 2D image gives information about the body part 202 from its viewpoint. Thus, the anatomical 3D shape AS is constructed incrementally, and the more 2D images the intraoperative data ID contains, the more accurate the reconstructed anatomical 3D shape AS, as shown in the sequence of partial anatomical 3D shapes in the bottom part of Fig. 6.

[0076] In addition to steps S32 and S34 described with reference to FIG. 3, according to a further embodiment illustrated in the flow chart of FIG. 7, in a further step S36, the anatomical 3D shape AS init The initial reconstruction of the anatomical 3D shape A is further improved using the unsegmented imaging data. Given potential errors in the calibration and segmentation, the reconstructed initial anatomical 3D shape A init To improve the quality of the 3D shape, a 3D shape refinement model is utilized. The 3D shape refinement model, in particular a convolutional neural network CNN architecture, has two input streams. The first input stream is the anatomical 3D shape AS init The second input stream includes a 2D segmentation of a particular body part 202 on the 2D image of the intraoperative imaging data ID. In this way, the patient-specific geometric information carried in the original intraoperative 2D image is infused, thereby forming an improved anatomical 3D shape AS enh By reconstructing the initial reconstruction, a 3D shape refinement model is trained to complete the missing components of the initial reconstruction (assuming possible data loss in the initial reconstruction process due to missing projection views).

[0077] 8, one embodiment for determining the predetermined position 5p of the instrument 5 is described with reference to the use case of a surgical procedure to implant pedicle screws into the vertebrae of a patient 200. The predetermined position 5p of the instrument 5 is determined by an artificial intelligence based optimization function based on the anatomical 3D shape AS of a particular body part 202 as well as data indicative of the surgical procedure.

[0078] Determining the predetermined position 5p of the instrument 5 based on the anatomical 3D shape AS is advantageous as it has the potential to improve the surgical workflow by abandoning the need for intraoperative scanning and corresponding manual processes, which can be costly and time-consuming. Supervised learning and reinforcement learning RL are used to train an artificial intelligence-based optimization function based on a clinical dataset consisting of ideal screw trajectories identified by experts. The predetermined position 5p of the instrument 5 is then determined based on the ideal screw trajectories IST and further based on prior knowledge of the geometry of the instrument 5.

[0079] 9A, 9B, and 9C show embodiments of positioning guidance data GD, where Fig. 9A shows the positioning guidance data GD including a visual representation of an estimated current position 5c of the instrument 5 and a visual representation of a predetermined position 5p of the instrument 5 superimposed on a visual representation of a reconstructed anatomical 3D shape AS.

[0080] FIG. 9B shows the positioning guidance data GD including a visual representation of the estimated current position 5c of the instrument 5 superimposed on a 2D image of the intraoperative imaging data ID.

[0081] FIG. 9C shows positioning guidance data GD including a visual representation of the estimated current position 5c of the instrument 5, a visual representation of the predetermined position 5p of the instrument 5, and a visual representation of the ideal screw trajectory of the surgical implant on a visual representation of the reconstructed anatomical 3D shape AS. [Explanation of symbols]

[0082] 1 System 2 Operating table 5. Equipment 5c Current location of the device 5p Equipment in place 10. Computing Devices 12 Data output interface (of a computing device) 14 Data input interface (of a computing device) 16 Processing unit (of a computing device) 18 Storage unit (of a computing device) 20 Imaging Device 22 Generator (X-ray source) 24 C-shaped connection element (C-arm) 26 Detector 30 Display Devices 32 Computer Screen Configuration 200 patients 202 (Patient) specific body part ID Intraoperative imaging data GD Positioning Guidance Data AS Anatomical 3D shape IST Ideal pedicle screw trajectory P 1-n Viewpoint (of 2D intraoperative images)

Claims

1. 1. A computer-implemented method for assisting in positioning an instrument (5) relative to a particular body part (202) of a patient (200), comprising: a) receiving, by a computing device (10), intraoperative imaging data (ID) from an imaging device (20) positioned near the patient (200), the intraoperative imaging data (ID) including 2D images, a plurality of the 2D images imaging the particular body part (202) of the patient (200) from a plurality of different viewpoints relative to the particular body part (202) of the patient (200), one or more of the plurality of 2D images imaging at least a portion of the instrument (5) from at least one viewpoint; b) using the computing device (10), reconstructing an anatomical 3D shape (AS) of the specific body part (202) based on the intraoperative imaging data (ID) and data indicating viewpoints corresponding to the plurality of 2D images using an artificial intelligence-based algorithm corresponding to the specific body part (202); c) estimating, by the computing device (10), a current position (5c) of the instrument (5) relative to the anatomical 3D shape (AS) of the specific body part (202) based on the intraoperative imaging data (ID); d) generating, by said computing device (10), positioning guidance data (GD) comprising a visual representation of said estimated current position (5c) of said instrument (5) relative to said anatomical 3D shape (AS) of said particular body part (202); A method comprising:

2. 2. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to claim 1, comprising: and the step of reconstructing the anatomical 3D shape (AS) of the specific body part (202) by the computing device (10) comprises training the artificial intelligence-based algorithm using a number of annotated imaging datasets of body parts of the patient (200) corresponding to the specific body part (202); A computer-implemented method, wherein the annotations include data identifying and / or describing characteristics of the body part.

3. 3. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to claim 2, comprising: A computer-implemented method comprising generating a number of annotated imaging data sets from annotated 3D imaging data, in particular computed tomography (CT) scans, of a body part of the patient (200) corresponding to the particular body part (202), the number of annotated imaging data sets including 2D images of the body part of the patient (200) corresponding to the particular body part (202) from a number of different viewpoints.

4. 4. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to one of claims 1 to 3, comprising: The step of reconstructing the anatomical 3D shape (AS) comprises: a) segmenting the intraoperative imaging data (ID) to identify the specific body part (202) of the patient (200); b) further using the segmented intraoperative imaging data (ID) to reconstruct the anatomical 3D shape (AS); 11. A computer-implemented method comprising:

5. 5. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to claim 4, comprising: The step of segmenting the intraoperative imaging data (ID) comprises: a) identifying a region of interest in the intraoperative imaging data (ID) comprising the particular body part (202) of the patient (200) using an artificial intelligence-based detection and segmentation model, such as a convolutional neural network-based detection and segmentation model; b) semantically segmenting the region of interest using the artificial intelligence based detection and segmentation model, thereby generating the segmented intraoperative imaging data (ID); 11. A computer-implemented method comprising:

6. 4. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to one of claims 1 to 3, comprising: The computer-implemented method further comprises estimating, by the computing device (10), the viewpoint corresponding to the intraoperative imaging data (ID).

7. 7. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to claim 6, comprising: the step of estimating a viewpoint corresponding to the intraoperative imaging data (ID) is performed using an instrument geometric model representing the geometry of the instrument (5); a) calculating a plurality of projections of said instrument geometric model from a plurality of candidate viewpoints; b) identifying the viewpoint corresponding to the intraoperative imaging data (ID) by comparing the at least part of the instrument (5) imaged in the 2D image of each of the intraoperative imaging data (ID) with the plurality of projections from the plurality of candidate viewpoints; 11. A computer-implemented method comprising:

8. 4. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to one of claims 1 to 3, comprising: the step of estimating the current position (5c) of the instrument (5) is performed using an instrument geometric model representing the geometry of the instrument (5); The step of estimating the current position (5c) of the instrument (5) comprises: a) comparing a projection of said instrument geometric model onto one or more planes of said 2D images of said intraoperative imaging data (ID) with said at least a portion of said instrument (5) imaged in each said 2D image of said intraoperative imaging data (ID), and / or b) determining the position of said instrument geometric model that generates a projection onto said plane of said 2D images of said intraoperative imaging data (ID) that matches said at least part of said instrument (5) imaged in each said 2D image of said intraoperative imaging data (ID); 11. A computer-implemented method comprising:

9. 4. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to one of claims 1 to 3, comprising:

1. A computer-implemented method, wherein generating positioning guidance data (GD) comprises superimposing the visual representation of the estimated current position (5c) of the instrument (5) onto a visual representation of the reconstructed anatomical 3D shape (AS).

10. 4. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to one of claims 1 to 3, comprising: a) identifying, by said computing device (10), a predetermined position (5p) of said instrument (5) relative to said anatomical 3D shape (AS) of said particular body part (202); b) superimposing a visual representation of the predetermined position (5p) of the instrument (5) onto a visual representation of the estimated current position (5c) of the instrument (5); The computer-implemented method further comprising:

11. 11. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to claim 10, comprising: said step of identifying said predetermined position (5p) of said instrument (5) a) retrieving, by said computing device (10), said predetermined position (5p) of said instrument (5) from a data store contained in or communicatively connected to said computing device (10); and / or b) calculating, by said computing device (10), said predetermined position (5p) of said instrument (5) determined by an optimization function based on said anatomical 3D shape (AS) of said body part and on data describing a surgical procedure; 11. A computer-implemented method comprising:

12. 4. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to one of claims 1 to 3, comprising: a) providing an instrument (5) according to an instrument geometric model, and / or b) capturing intraoperative imaging data (ID) using an imaging device (20) that images at least a body part of said patient (200) and at least a part of said instrument (5); and / or c) controlling, by said computing device (10), a display device (30) to display at least a portion of said guidance data (GD). The computer-implemented method further comprising:

13. 4. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to one of claims 1 to 3, comprising: The intraoperative imaging data (ID) a) radiation-based images, in particular X-ray images, of the particular body part (202) of the patient (200) and of a part of the instrument (5), respectively; b) ultrasound images of the particular body part (202) of the patient (200) and of a portion of the instrument (5), respectively; c) respective arthroscopic images of said particular body part (202) of said patient (200) and a portion of said instrument (5); d) optical images of the particular body part (202) of the patient (200) and a portion of the instrument (5), respectively; e) cross-sectional imaging of a portion of said patient (200) and said instrument (5), respectively; A computer-implemented method comprising one or more of:

14. 4. A computer-implemented method for assisting in positioning an instrument (5) relative to a specific body part (202) of a patient (200) according to one of claims 1 to 3, comprising:

1. A computer-implemented method, wherein the steps of receiving intraoperative imaging data (ID), estimating a current position (5c) of the instrument (5), and generating guidance data (GD) are performed repeatedly or continuously over a period of time in preparation for / prior to a surgical treatment of the patient (200).

15. a) a data input interface (14) communicatively connectable to an imaging device (20) and configured to receive intraoperative imaging data (ID); b) a data output interface (12) configured to transmit at least a portion of the guidance data (GD) to a display device (30) communicatively connectable to said data output interface (12); b) a processing device (16); c) a storage unit (18) containing instructions which, when executed by the processing unit (16), cause the computing device (10) to perform the method of any one of claims 1 to 3; and A computing device (10) comprising:

16. A system (1) for assisting in the positioning of an instrument (5) relative to a particular body part (202) of a patient (200), comprising: a) a computing device (10) according to claim 15; b) an imaging device (20) arranged and configured to capture intraoperative imaging data (ID) comprising 2D images, wherein the intraoperative imaging data (ID) comprises 2D images, a plurality of the 2D images capturing the particular body part (202) of the patient (200) from a plurality of different viewpoints relative to the particular body part (202) of the patient (200), and one or more of the plurality of 2D images capturing at least a portion of the instrument (5) from at least one viewpoint; c) a display device (30) configured to display a human-interpretable representation of at least a portion of said guidance data (GD), said display device (30) communicatively connected to said data output interface (12) of said computing device (10); Equipped with A system (1) configured to carry out the method according to any one of claims 1 to 3.

17. 17. A system (1) for assisting in positioning an instrument (5) relative to a particular body part (202) of a patient (200) as claimed in claim 16, further comprising an instrument (5) having a geometry corresponding to the instrument geometric model.

18. 4. A computer program product comprising instructions that, when executed by a processing unit (16) of a computing device (10), cause the computing device (10) to perform the method of any one of claims 1 to 3.

19. 20. The computer program product of claim 18, wherein the instructions include an artificial intelligence based algorithm corresponding to a particular body part of a patient, the artificial intelligence based algorithm being trained using a number of annotated imaging datasets of body parts corresponding to the particular body part of the patient, the annotations including data identifying and / or describing characteristics of the body part.

Images