Computer-implemented methods, devices, systems, and computer program products for processing anatomical imaging data

Abstract

1. A computer-implemented method for processing anatomical imaging data, the method comprising: acquiring, for a plurality of anatomical specimens, 3D imaging data of a particular body part of the anatomical specimens among the plurality of anatomical specimens; acquiring 2D images of the particular body part of the anatomical specimens from a plurality of different viewpoints relative to the particular body part; determining a plurality of camera matrices P corresponding to the acquired 2D images; generating a composite 2D image by projecting the acquired 3D imaging data according to the plurality of camera matrices P; and associating the acquired 2D image with one of the composite 2D images as a training data pair according to the plurality of camera matrices P and the plurality of viewpoints. The method further comprises generating a training data set for use in training an artificial intelligence algorithm using the training data pairs.

Description

[Technical Field]

[0001] The present disclosure relates to computer-implemented methods, devices, systems, and computer program products for processing anatomical imaging data. In particular, the present disclosure relates to computer-implemented methods, devices, systems, and computer program products for processing anatomical imaging data, including generating a training data set for use in training an artificial intelligence algorithm. Furthermore, the present disclosure relates to computer-implemented methods, devices, systems, and computer program products for processing anatomical imaging data, including training an artificial intelligence algorithm using the training data set. Furthermore, the present disclosure relates to computer-implemented methods, devices, systems, and computer program products for processing anatomical imaging data, including processing acquired 2D images of specific body parts of an anatomical object using an artificial intelligence algorithm.

[0002] The present invention further relates to a computer-implemented method for assisting in the positioning of an instrument, such as a surgical instrument, relative to a particular body part of a patient.The present invention further relates to a computing device configured to assist in the positioning of an instrument, such as a surgical instrument, relative to a particular body part of a patient.The present invention further relates to a system for assisting in the positioning of an instrument relative to a particular body part of a patient.The present invention further relates to a computer program product comprising instructions which, when executed by a processing unit of a computing device, cause the computing device to assist in the positioning of an instrument, such as a surgical instrument, relative to a particular body part of a patient. [Background technology]

[0003] Processing anatomical imaging data, especially for intraoperative navigation and guidance of complex orthopedic interventions, has thus far remained a challenging task despite the existence of modern computer-assisted surgery (CAS) solutions. This is reflected in recent reports that estimated clinical adoption rates of CAS solutions to be as low as 11% and 5% of performed surgeries. This is particularly sobering in light of the established evidence regarding the benefits of CAS solutions in improving surgical accuracy and outcomes. Historically, the relatively low clinical adoption rates of CAS solutions have been attributed to many factors, ranging from extra surgical time and operational costs due to line-of-sight issues (endemic to solutions requiring optical tracking systems) to, most importantly, the requirement for patient registration.

[0004] Patient registration, the process of aligning a preoperatively generated surgical plan to the patient's anatomy, has been addressed in the prior art using a variety of algorithms, including feature-based registration, intensity-based registration, statistical shape modeling (SSM), and more recently, data-driven artificial intelligence registration methods. Despite the availability of advanced algorithms for patient registration, commercially available methods for processing anatomical imaging data, particularly CAS solutions, generally rely on rudimentary techniques such as landmark-based methods. This is because, despite their low level of autonomy and generality, such basic methods can be easily integrated into CAS pipelines to some extent, in contrast to advanced registration methods, which suffer from limiting factors such as small capture ranges, lack of large training datasets, lack of robust similarity metrics for multimodality registration, and long computational times.

[0005] This has been the reason behind the emergence of algorithms aimed at providing registration-free alternatives for processing anatomical imaging data. In the context of orthopedic surgery, such methods typically rely solely on widely available 2D imaging data, e.g., in the form of C-arm fluoroscopy. As an essential component, these alternative solutions require processing anatomical imaging data, e.g., C-arm imaging, to directly reconstruct the anatomy of an anatomical object based solely on 2D imaging data. Registration-free CAS can be achieved by full intraoperative cone-beam computed tomography (CBCT) or by more advanced algorithms that can reconstruct the 3D shape of a patient's anatomy using sparse fluoroscopy data. Registration-free CAS solutions based on intraoperative CBCT imaging have not been widely adopted due to reasons such as extra cost, time, and ionizing radiation.

[0006] Artificial intelligence-based algorithms designed to process anatomical imaging data, particularly anatomical 3D reconstructions based on x-ray input, do not suffer from the aforementioned limitations due to their ability to process anatomical imaging data, particularly in reconstructing patient anatomy based on a small number of widely available 2D images, especially intraoperative x-rays. These artificial intelligence-based data-driven methods can be understood as a stepping stone toward the development of registration-free methods for processing anatomical imaging data, such as future CAS systems.

[0007] However, the main technical bottleneck in creating such methods is the limited availability of large-scale annotated anatomical imaging. Among other factors, ethical and radiation exposure concerns, along with the manual effort required for data annotation, are fundamental obstacles in collecting the required in vivo training datasets. To overcome these obstacles, the majority of existing studies opt to create synthetic 2D datasets (e.g., X-rays) that can be automatically generated with corresponding annotations. These synthetic datasets are typically created using digital reconstruction radiography (DRR) techniques, which can simulate planar X-ray projections based on input CT volumes with varying degrees of realism. Despite the flexibility and scalability of such datasets, they may fail to capture realistic anatomical imaging conditions, such as the actual radiological properties of tissues, the noise characteristics of imaging devices, and the backscatter behavior of real 2D imaging environments. When used to train downstream artificial intelligence models, such as 3D reconstruction networks, these synthetic datasets inevitably tune the models to perform well on the synthetic training data alone but fail to achieve similar levels of performance when applied to real (acquired) 2D images (e.g., X-ray images). This is commonly referred to as the domain gap phenomenon. Summary of the Invention [Problem to be solved by the invention]

[0008] It is an object of the present disclosure to provide a computer-implemented method, computing device, and computer program product, respectively, for processing anatomical imaging data, including generating training data sets for training artificial intelligence algorithms, that do not have at least some of the drawbacks of the prior art.

[0009] In particular, it is an object of the embodiments disclosed herein to provide a computer-implemented method, computing device, and computer program product, respectively, for processing anatomical imaging data using a training data set comprising paired real-synthetic X-ray images that may be acquired from an anatomical specimen. [Means for solving the problem]

[0010] According to the present disclosure, these objects are addressed by the features of independent claims 1, 14, 15 and 16. Furthermore, further advantageous embodiments emerge from the dependent claims and the description.

[0011] This object is addressed, inter alia, by a computer-implemented method for processing anatomical imaging data, the method comprising the steps of: acquiring, for a plurality of anatomical specimens, 3D imaging data of particular body parts of the anatomical specimens among the plurality of anatomical specimens (referred to as acquired 3D imaging data); acquiring 2D images of the particular body parts of the anatomical specimens from a plurality of different viewpoints relative to the particular body parts (referred to as acquired 2D images); determining a plurality of camera matrices P corresponding to the acquired 2D images, whereby the camera matrix P indicates the viewpoints of the 2D images relative to at least the particular body parts; generating a composite 2D image by projecting the acquired 3D imaging data according to the plurality of camera matrices P; and associating the acquired 2D image with one of the composite 2D images as a training data pair according to the plurality of camera matrices P. The method further includes using the training data pairs to generate a training data set for use in training an artificial intelligence algorithm for processing the anatomical imaging data. In particular, generating the training data set using the training data pairs includes storing the data pairs along with corresponding metadata and / or data indicating the association between the acquired 2D image and the composite 2D image.

[0012] According to embodiments, one or more of the anatomical specimens are cadaveric, in particular human cadaveric specimens. Alternatively or additionally, one or more of the anatomical specimens are living, in particular human living specimens (persons).

[0013] According to embodiments, acquiring 3D imaging data of a particular body part of an anatomical specimen from the plurality of anatomical specimens includes capturing the particular body part of the anatomical specimen with a 3D imaging device, such as a computed tomography CT imaging device. Alternatively or additionally, acquiring 3D imaging data of the particular body part of the anatomical specimen from the plurality of anatomical specimens includes receiving, by the computing device, 3D imaging data (capturing the particular body part of the anatomical specimen) from a communicatively connected 3D imaging device. Alternatively or additionally, acquiring 3D imaging data of the particular body part of the anatomical specimen from the plurality of anatomical specimens includes retrieving, by the computing device, the 3D imaging data from a communicatively connected database that stores 3D imaging data of the plurality of anatomical specimens.

[0014] According to embodiments, acquiring 2D images of the particular body part of the anatomical specimen from a plurality of different viewpoints relative to the particular body part includes capturing, by a 2D imaging device, 2D images of the particular body part of the anatomical specimen from a plurality of different viewpoints relative to the particular body part. Alternatively or additionally, acquiring 2D images of the particular body part of the anatomical specimen from a plurality of different viewpoints relative to the particular body part includes receiving, by a computing device, 2D images from a communicatively connected 2D imaging device. Alternatively or additionally, acquiring 2D images of the particular body part of the anatomical specimen from a plurality of different viewpoints relative to the particular body part includes retrieving, by a computing device, 2D images from a communicatively connected database and storing the 2D images of the particular body part of the anatomical specimen from a plurality of different viewpoints relative to the particular body part.

[0015] Determining the multiple camera matrices P corresponding to the acquired 2D images includes determining the intrinsic and extrinsic camera properties of the acquired 2D images. Intrinsic camera properties relate to properties that characterize the 2D imaging device that captured the 2D images, such as focal length, principal point location on the image plane, etc. Extrinsic camera properties relate to the position and orientation of the camera relative to a particular body part, in other words, the camera's viewpoint.

[0016] Having determined a camera matrix P corresponding to the acquired 2D images, which indicates the viewpoint of the 2D images relative to at least a particular body part, a composite 2D image is generated by a computing device by projecting the acquired 3D imaging data according to the camera matrices P. In other words, the composite 2D image is generated as if it were captured by the same 2D imaging device from the same viewpoint (same camera matrix) as the acquired 2D images. The technical consideration behind this approach is that due to unavoidable imprecision in intrinsic and extrinsic camera properties, it is nearly impossible to precisely match the camera matrix P with a prescribed camera matrix (of the composite 2D image) when acquiring a 2D image of an anatomical specimen. Instead of attempting to correct for the imprecision in setting the intrinsic and extrinsic camera properties when capturing a 2D image of an anatomical specimen, in accordance with the present invention, generation of a composite 2D image that does not suffer from such imprecision is performed according to the camera matrix of the acquired 2D images.

[0017] In other words, instead of attempting to position the device in the 2D image to match the camera matrix, and in particular the viewpoint of the synthesized 2D image, when generating the training data pairs, the 2D image is synthesized according to the camera matrix P of the acquired 2D image.

[0018] The training data set generated using the training data pairs is specifically adapted for use in training an artificial intelligence algorithm for processing anatomical imaging data. The training data set is generated taking into account specific technical considerations to provide training data pairs, whereby acquired 2D images are associated with a composite 2D image according to a corresponding camera matrix P. In particular, one or more of the acquired 2D images are associated with a corresponding camera matrix P, particularly a composite 2D image generated with a corresponding viewpoint. In this way, the training data pairs (including related acquired and composite 2D images) are not only appropriate but also specifically adapted for use in training an artificial intelligence algorithm in that each data pair includes an acquired 2D image provided as input and a composite 2D image as a reference for the expected / desired output, or vice versa, i.e., a composite 2D image as input and an acquired 2D image as the expected / desired output. By providing corresponding data pairs, the artificial intelligence algorithm can be trained by applying an appropriate objective function, cost function, or loss function that compares the generated output with the desired output.

[0019] To enable precise determination of the multiple camera matrices P corresponding to the acquired 2D images and to enable projection of the acquired 3D imaging data according to the multiple camera matrices P, according to an embodiment, the anatomical specimen is provided (fitted) with a calibration device before acquiring 3D imaging data and 2D images of a specific body part of the anatomical specimen. In the case of a live anatomical specimen, a registration device is used as the calibration device that is non-invasively or minimally invasively attached to the anatomical specimen. In the case of an ex vivo cadaveric anatomical specimen, a reference device is attached to the anatomical specimen as the calibration device.

[0020] Upon providing the calibration device to the anatomical specimen, the 2D coordinates of the calibration device are determined in each of the 2D images (referred to as determined 2D coordinates). Furthermore, once the calibration device is provided to the anatomical specimen, 3D coordinates of the calibration device are determined based on the 3D imaging data. Then, 2D-3D coordinate correspondences are determined between the 2D coordinates of the calibration device in each of the 2D images and the 3D coordinates of the calibration device. The 2D-3D coordinate correspondences are then used to further determine multiple camera matrices P corresponding to the acquired 2D images. In other words, by determining the coordinates of the calibration device in both the acquired 2D and 3D images, the camera matrix P can be precisely determined, and the 3D image data can be projected to generate a composite 2D image that precisely matches the camera matrix P of the acquired 2D images.

[0021] According to an embodiment, to prevent the calibration device from being captured in the 3D imaging data and thus projected onto the composite 2D image, 3D imaging data is further acquired before providing the calibration device to the anatomical specimen. To compensate for resulting registration errors, a computing device performs 3D-3D registration of the 3D imaging data acquired before providing the calibration device to the anatomical specimen and the 3D imaging data acquired after providing the calibration device to the anatomical specimen. After 3D-3D registration of the 3D imaging data acquired before providing the calibration device to the anatomical specimen and the 3D imaging data acquired after providing the calibration device to the anatomical specimen, a composite 2D image is generated by projecting the 3D imaging data acquired before providing the calibration device to the anatomical specimen according to the multiple camera matrices P. In this way, the calibration device is not present in the composite 2D image (it is only present in the anatomical specimen).

[0022] While the calibration device is useful for determining the camera matrix P of the acquired 2D images, the presence of the calibration device in the acquired 2D images is undesirable. Therefore, to remove the calibration device captured in the 2D imaging data, according to an embodiment, the projection of the calibration device in the 2D images is restored from the acquired 2D images before being used in generating the training data set.

[0023] According to an embodiment, determining multiple camera matrices P (corresponding to the acquired 2D images) includes detecting 2D coordinates of the calibration device on the acquired 2D images, detecting 3D coordinates of the calibration device on the acquired 3D imaging data, determining 2D-3D coordinate correspondences between the 2D and 3D coordinates of the calibration device, and determining the camera matrix P using the 2D and 3D coordinates of the calibration device and the 2D-3D coordinate correspondences. In particular, determining the 2D-3D coordinate correspondences between the 2D and 3D coordinates of the calibration device is performed in an iterative process, whereby for each possible correspondence between the 2D and 3D coordinates of the calibration device, a camera matrix P is estimated, the 3D imaging data is projected according to the estimated camera matrix P, and a reprojection error is calculated by comparing the projected 3D imaging data with the corresponding acquired 2D image. After calculating the reprojection error for all possible correspondences between the 2D and 3D coordinates of the calibration device, the 2D-3D coordinate correspondence with the lowest reprojection error is used to determine the camera matrix P for the acquired 2D images.

[0024] According to an embodiment, the computer-implemented method of the present invention further includes obtaining a pre-training dataset for processing anatomical imaging data, the pre-training dataset including multiple synthetic 2D images capturing specific body parts of an anatomical specimen, and pre-training an artificial intelligence algorithm (referred to as a pre-trained artificial intelligence algorithm) using the pre-training dataset for processing anatomical imaging data. Given the limited availability of annotated anatomical imaging, according to certain embodiments, the pre-training dataset includes synthetic 2D images, such as fluoroscopy shots (i.e., DRRs), generated with various camera matrices, particularly from various viewpoints, given input 3D image data, such as CT scans. For example, using this method, up to hundreds of annotated "synthetic" 2D images (capturing specific body parts) can be generated from a single annotated CT scan, which can be used to train an artificial intelligence algorithm for processing anatomical images, such as reconstructing accurate anatomical 3D shapes from as few 2D intraoperative images as possible.

[0025] In light of the limitations and drawbacks of the domain gap as described in the Background section of this application, a further objective of the embodiments disclosed herein is to provide a domain adaptation framework method that may be used to utilize artificial intelligence algorithms pre-trained only on synthetic images for processing acquired (real) 2D images (e.g., X-rays). For example, such a method may be used to reconstruct the anatomical 3D shape (AS) of a particular body part of an anatomical object based on the 2D image of the particular body part and, in particular, data indicating a viewpoint corresponding to the camera matrix 2D image, and / or for anatomical segmentation of the 2D image of the particular body part of the anatomical object.

[0026] This objective is achieved by applying a transfer learning process, i.e., by training (also called retraining) a pre-trained artificial intelligence algorithm using a training dataset containing data pairs of synthetic 2D images and acquired (real) 2D images of the corresponding camera matrix P.

[0027] According to embodiments, training (also referred to as retraining) the pre-trained artificial intelligence algorithm includes attaching a first subset of network parameters, such as artificial neural network weights and biases, for the network components of the pre-trained artificial intelligence algorithm and adjusting a second subset of the network parameters using the training data pairs of the training dataset as inputs. In a specific embodiment for reconstructing the 3D anatomical shape of a particular body part of an anatomical object, retraining the pre-trained artificial intelligence algorithm includes attaching network parameters, such as artificial neural network weights and biases, for the network components of the pre-trained artificial intelligence algorithm, except for the 2D feature extractor module, and adjusting the network parameters of the 2D feature extractor module using the training data pairs of the training dataset as inputs.

[0028] In practical application, according to the present invention, the computer-implemented method further includes acquiring 2D images of a particular body part of an anatomical object, and processing the acquired 2D images using an artificial intelligence algorithm trained using a training data set.

[0029] In this context, the term "anatomical object" refers to a particular object of interest, such as a patient undergoing surgery or medical examination, rather than multiple "anatomical specimens" that are analyzed to obtain a training dataset.

[0030] Alternatively, the objective of providing a domain adaptation framework may be addressed by a style transfer process, which utilizes an artificial intelligence algorithm pre-trained on synthetic images for processing acquired (real) 2D images (e.g., X-rays). In particular, the objective of providing a domain adaptation framework is addressed, according to an embodiment, in that the computer-implemented method further includes training a style transfer algorithm using a training dataset and applying the style transfer process to the acquired 2D images using the style transfer algorithm. The style transfer process is applied to the acquired 2D images (referred to as pre-conditioned acquired 2D images) prior to processing the acquired 2D images of the anatomical object using the pre-trained artificial intelligence algorithm.

[0031] In other words, the domain gap (pre-training performed using widely available but synthetic 2D images) is bridged in that the input images (acquired 2D images of anatomical objects) are pre-conditioned by a style transfer algorithm before being fed as input into the pre-trained artificial intelligence algorithm.

[0032] Specifically, the style transfer process involves extracting texture information from synthetic 2D images of a training dataset and transferring the texture information onto acquired 2D images of anatomical objects while preserving the underlying semantic content of the acquired 2D images. In other words, the acquired 2D images (of anatomical objects) are made to look like the synthetic 2D images that the pre-trained artificial intelligence algorithm was trained to process.

[0033] Unique to the style transfer process of the present invention is the development of a model that transforms acquired 2D images (e.g., real X-ray data) of anatomical objects into synthetic 2D images (e.g., synthetic DRR images). Compared to known approaches that transfer synthetic 2D images (e.g., synthetic DRR images) to acquired (e.g., realistic X-ray-like) images when the target domain is highly heterogeneous, the inverse transfer direction of the present invention is implemented to ensure a homogeneous target domain (e.g., by controlling the synthetic DRR space). Furthermore, in various applications where pre-trained artificial intelligence algorithms on synthetic data are already available (e.g., segmentation, reconstruction, landmark detection, etc.), the style transfer process of the present invention can be used sequentially, where the acquired 2D image (e.g., real X-ray) input is first transferred into the synthetic domain, which is then fed into the pre-trained artificial intelligence algorithm.

[0034] In certain embodiments, the style transfer process implements one or more of the following methods: Encoder-Decoder Deep Convolutional Network EDDC, Neural Style Transfer NST Algorithm, Deep Convolutional Generative Adversarial Network (DCGAN), Cycle-consistent generative adversarial networks, CycleGAN.

[0035] As a practical application, according to the present invention, the computer-implemented method further includes acquiring 2D images of a particular body part of an anatomical object, applying a style transfer process to the acquired 2D images using a style transfer algorithm, and processing the pre-conditioned acquired 2D images using a pre-trained artificial intelligence algorithm.

[0036] As a side note, in the context of this disclosure, the terms "pre-training" and "pre-trained" refer to the training of an artificial intelligence algorithm using training data other than the training dataset generated as part of the present invention.

[0037] As a first use case, according to an embodiment of the present invention, processing acquired 2D images of a particular body part of an anatomical object using an artificial intelligence algorithm includes reconstructing an anatomical 3D shape of the particular body part of the anatomical object based on the acquired 2D images of the particular body part and data indicating a viewpoint corresponding to the acquired 2D images.

[0038] As a further use case, according to an embodiment of the present invention, processing acquired 2D images of a particular body part of an anatomical object using an artificial intelligence algorithm includes anatomically segmenting the acquired 2D images to identify one or more anatomical features of the particular body part.

[0039] The present application further relates to a computing device comprising a processing unit configured to perform the method according to one of the embodiments disclosed herein. According to the embodiments, the computing device comprises any one or combination of a central processing unit CPU, a physical or virtual computer, a local and / or remote portable, desktop and / or server computer, a cloud-based server or software as a service, and / or dedicated hardware circuitry such as an ASIC or FPGA.

[0040] The present application further relates to a system comprising a computing device according to one of the embodiments disclosed herein and an imaging device, such as a pre-operative, post-operative, and / or intra-operative imaging device, communicatively connected to the computing device and configured to capture 2D images of a particular body part of an anatomical object.

[0041] The present application further relates to a computer program product comprising instructions that, when executed by a processing unit of a computing device, cause the computing device to perform a method according to any one of the embodiments disclosed herein.

[0042] According to the present invention, the training data sets, pre-trained artificial intelligence algorithms, trained artificial intelligence algorithms, and / or style transfer algorithms are each generated and trained for a specific body part, and correspondingly, their use for training and processing, respectively, acquired 2D images for the same specific body part.

[0043] Another object of the present invention is to provide a method, computing device, system, and computer program product for assisting in positioning an instrument relative to a particular body part of a patient that overcomes one or more of the drawbacks of the prior art.

[0044] In particular, it is an object of the present invention to provide a registration-free method for assisting in the positioning of an instrument relative to a particular body part of a patient that can reconstruct anatomical 3D geometry and generate a visual representation of the instrument's position relative to the particular body part of the patient using only intraoperative 2D imaging data, i.e., without requiring navigation hardware to be installed in the operating room.

[0045] According to the present disclosure, this object is addressed by the features of independent claim 1. Further advantageous embodiments also emerge from the dependent claims and the description. In particular, the object is to provide a computer-implemented method for assisting in the positioning of an instrument, such as a surgical instrument (e.g., a surgical drill, knife, or surgical laser instrument) or a medical diagnostic instrument, relative to a particular body part of a patient, comprising: receiving intraoperative 2D imaging data; reconstructing an anatomical 3D shape using the intraoperative 2D imaging data and using artificial intelligence algorithms corresponding to the specific body part; estimating a current position of the instrument based on intraoperative 2D imaging data; generating positioning guidance data including a visual representation of an estimated current position of the instrument relative to the anatomical 3D shape of the particular body part; This is achieved by a method comprising:

[0046] In certain embodiments, the steps of receiving intraoperative 2D imaging data, estimating the current position of the instrument, and generating guidance data are performed repeatedly or continuously over a period of time in preparation for / prior to surgical treatment of the patient. Receiving intraoperative 2D imaging data Intraoperative 2D imaging data is received by a computing device from an intraoperative imaging device positioned near a patient. As used herein, positioning the intraoperative imaging device near a patient refers to a positioning that allows the intraoperative imaging device to capture intraoperative 2D imaging data of the patient. The imaging data includes a plurality of intraoperative 2D images. Two or more of the intraoperative 2D images capture a particular body part of the patient from two or more different viewpoints relative to the particular body part of the patient. One or more of the same plurality of intraoperative 2D images capturing a particular body part also capture at least a portion of an instrument from at least one viewpoint. In the context of the present invention, the term viewpoint, with respect to the viewpoint of the imaging data, refers to the position and / or orientation (e.g., roll, pitch, yaw) (e.g., in an x, y, and z Cartesian coordinate system) of the intraoperative imaging device relative to the particular body part and relative to the instrument, respectively.

[0047] According to embodiments of the present invention, the intraoperative 2D imaging data comprises one or more of: a) radiation-based images, in particular X-ray images; b) ultrasound images; c) arthroscopic images; d) optical images; and / or e) any other cross-sectional images.

[0048] Images of specific body parts of the patient and parts of the instruments are captured using an intraoperative imaging device communicatively connected to a computing device. In the case of radiation-based images, the intraoperative imaging device may include a C-arm intraoperative imaging device based on X-ray technology. The C-arm intraoperative imaging device includes a generator (X-ray source) and an image intensifier or flat-panel detector. A C-shaped connecting element allows movement horizontally, vertically, and / or around a swivel axis, so that 2D X-ray images of the patient can be generated from various viewpoints around the patient. The generator emits X-rays that penetrate the patient's body. The image intensifier or detector converts the X-rays into a visible image, which is transmitted to the computing device.

[0049] The intraoperative 2D imaging data includes data indicative of viewpoints corresponding to the plurality of intraoperative 2D images, specifying the position and / or orientation of an intraoperative imaging device capturing the plurality of intraoperative 2D images, such as the position in an x, y, and z Cartesian coordinate system of the intraoperative imaging device relative to a particular body part, and / or its orientation as roll, pitch, and yaw. According to embodiments disclosed herein, the data indicative of viewpoints corresponding to the plurality of intraoperative 2D images is stored in a data store included in or communicatively connected to a computing device. Alternatively or additionally, the viewpoints corresponding to the intraoperative 2D imaging data are estimated by the computing device based on the intraoperative 2D imaging data.

[0050] In certain embodiments, estimating a viewpoint corresponding to the intraoperative 2D imaging data is performed using an instrument geometric model that represents the geometry of the instrument. First, multiple projections of the instrument geometric model are calculated from multiple candidate viewpoints. The candidate viewpoints are selected as distinct viewpoints within a predefined space of possible viewpoints of the intraoperative imaging device. In other words, unrealistic positions and orientations of the intraoperative imaging device relative to the patient are not considered to conserve computing power. Then, a viewpoint corresponding to an intraoperative 2D image of the intraoperative 2D imaging data is identified by comparing at least a portion of the instrument captured by each 2D image of the intraoperative 2D imaging data with multiple projections calculated from the multiple candidate viewpoints. In particular, the comparison includes applying a matching function to identify a best match between one of the calculated projections of the “virtual” instrument (based on the instrument geometric model) from the multiple candidate viewpoints and the portion of the “physical” instrument captured by the intraoperative imaging device. The candidate viewpoint that produces the best match is selected as the estimated viewpoint.

[0051] According to embodiments disclosed herein, estimating a viewpoint corresponding to intraoperative 2D imaging data is performed using an artificial intelligence algorithm trained using multiple imaging datasets with known viewpoints. To overcome limitations in the availability and / or accuracy of imaging datasets with known viewpoints, multiple imaging datasets containing intraoperative 2D images from known viewpoints are generated from 3D imaging data, particularly computed tomography (CT) scans. Using this pre-operatively trained artificial intelligence algorithm, the intraoperative position of an intraoperative imaging device can be estimated based solely on the intraoperative images, without the need for an external tracking device or calibration phantom. Anatomical 3D shape generation The anatomical 3D shape of the particular body part is reconstructed by the computing device based on the intraoperative 2D imaging data and data indicating viewpoints corresponding to the plurality of intraoperative 2D images using an artificial intelligence algorithm corresponding to the particular body part.

[0052] According to the embodiments disclosed herein, the anatomical 3D shape is reconstructed as a voxelized volume and / or mesh. It is important to emphasize that the artificial intelligence algorithm must be a model corresponding to the specific body part that allows the reconstruction of the 3D anatomical shape from multiple intraoperative 2D images of the specific body part.

[0053] According to certain embodiments disclosed herein, an artificial intelligence algorithm is trained using a number of annotated imaging datasets that capture body parts (potentially of persons other than the patient) that correspond to particular body parts of the patient. The annotations of the imaging datasets include data that identify and / or describe characteristics of the body parts, such as identifying pixels, vectors, contours, surfaces in intraoperative 2D images, and / or voxels in 3D images that capture the particular body part.

[0054] To overcome limitations in the availability and / or accuracy of annotated imaging datasets capturing body parts corresponding to specific body parts of a patient, according to embodiments disclosed herein, multiple annotated imaging datasets are generated from annotated 3D imaging data, particularly computed tomography (CT) scans, capturing body parts corresponding to specific body parts of a patient. In particular, given an input preoperative CT scan, synthetic intraoperative 2D images, such as fluoroscopy shots (i.e., DRRs), are generated from various viewpoints around the patient. For example, using this method, up to hundreds of annotated "synthetic" intraoperative 2D images (capturing specific body parts) can be generated from a single annotated CT scan, which can be used by artificial intelligence algorithms to improve the ability to reconstruct accurate anatomical 3D shapes from as few 2D intraoperative images as possible. Estimating the current location of the fixture Once the anatomical 3D shape is reconstructed, the current position of the instrument relative to the anatomical 3D shape of a particular body part is estimated based on the intraoperative 2D imaging data, particularly the intraoperative 2D images, of the imaging data capturing the instrument. According to the embodiments disclosed herein, estimating the current position of the instrument is performed using an instrument geometric model that represents the geometric shape of the instrument. First, a projection of the instrument geometric model is compared to at least a portion of the instrument captured by each 2D image of the intraoperative 2D imaging data. The instrument geometric model is projected onto one or more planes of the intraoperative 2D images of the intraoperative 2D imaging data capturing at least a portion of the instrument. The planes of the intraoperative 2D images of the intraoperative 2D imaging data are determined based on the viewpoints of each 2D image. Then, a position of the instrument geometric model is determined that generates a projection onto the plane of the intraoperative 2D images of the intraoperative 2D imaging data that (best) matches at least a portion of the instrument captured by each 2D image of the intraoperative 2D imaging data. In other words, a reverse process is applied compared to the (initial) determination of the viewpoint of the intraoperative 2D imaging data. However, this inverse process does not necessarily apply to the same intraoperative 2D images (of the intraoperative 2D imaging data) used to determine the image viewpoints used for reconstruction of the anatomical 3D shape.

[0055] According to the embodiments disclosed herein, at the initial stage of the method for assisting in instrument positioning, the anatomical 3D shape is reconstructed once, but estimation of the current position of the instrument is performed repeatedly at set intervals and / or triggered by certain events and / or manually triggered.

[0056] To improve the accuracy of estimating the instrument position and / or estimating the viewpoint corresponding to the intraoperative 2D imaging data, according to a further embodiment, the method of the present invention further includes providing the instrument according to an instrument geometric model. The instrument geometric model is specifically designed to optimize the estimation of the instrument position based on as few intraoperative images as possible. In particular, to enable estimation of the instrument orientation based on the intraoperative 2D images, the instrument is designed such that at least a portion of the instrument is not perfectly rotationally symmetric about any of the axes of a Cartesian coordinate system. Alternatively or additionally, the instrument is designed with a special marker to facilitate identification of the instrument based on the 2D intraoperative images. Generating positioning guidance data Upon reconstructing the anatomical 3D shape of the specific body part and estimating the current position of the instrument, positioning guidance data including a visual representation of the estimated current position of the instrument relative to the anatomical 3D shape of the specific body part is generated by the computing device. According to embodiments disclosed herein, the guidance data is generated as a 2D image displayed on a computer display. Alternatively or additionally, the guidance data is generated as an augmented reality overlay that enables an augmented reality device, such as a headset, to project an overlay onto a user's field of view, whereby the overlay is aligned with the user's viewpoint of the specific body part of the patient and / or includes overlay metadata aligned with the user's viewpoint of the instrument. According to embodiments disclosed herein, a visual representation of the estimated current position of the instrument is superimposed on the visual representation of the reconstructed anatomical 3D shape.

[0057] According to embodiments disclosed herein, the computing device controls a display device to display at least a portion of the guidance data, and the display device is a computer screen, an augmented reality headset, or any device configured to display the guidance data.

[0058] To guide the surgeon in correctly placing the instrument, according to further embodiments disclosed herein, a prescribed position of the instrument relative to the 3D anatomical shape of a particular body part is identified by a computing device, and a visual representation of the prescribed position of the instrument is superimposed on a visual representation of the estimated current position of the instrument. The prescribed position of the instrument is retrieved or received by the computing device from a data store contained in or communicatively connected to the computing device. Alternatively or additionally, the prescribed position of the instrument is calculated by the computing device, and the prescribed position of the instrument is determined by an optimization function based on data indicative of the 3D anatomical shape of the body part and the surgical procedure.

[0059] Advantageously, the embodiments disclosed herein enable automatic pre-operative planning based on the reconstructed anatomical 3D shape to guide the surgeon in the placement of surgical instruments. Given that intra-operative 2D imaging data is used to reconstruct the anatomical 3D shape of a body part (e.g., the spine), and a prescribed position / trajectory of the instrument can be identified based on the anatomical 3D shape of the body part, no pre-operative planning stage is required to define a safe implantation trajectory, and no registration of the pre-operative data to the intra-operative patient position is required.

[0060] Another object of the present invention is to provide a computing device for positioning of an instrument relative to a specific body part of a patient that can reconstruct anatomical 3D geometry and generate a visual representation of the position of the instrument relative to the specific body part of the patient using only intraoperative 2D imaging data, i.e., without requiring navigation hardware to be installed in the operating room and without requiring a preoperative planning registration process.

[0061] The above-identified objects are further achieved by a computing device including a data input interface, a data output interface, a processing device, and a storage unit. The data input interface, such as a wired (e.g., Ethernet, DVI, HDMI, VGA) and / or wireless data communication interface (e.g., 4G, 5G, Wifi, Bluetooth, UWB), is communicatively connected to the intraoperative imaging device and configured to receive intraoperative 2D imaging data from the intraoperative imaging device. The data output interface, such as a wired (e.g., Ethernet, DVI, HDMI, VGA) and / or wireless data communication interface (e.g., 4G, 5G, Wifi, Bluetooth, UWB, infrared), is configured to transmit at least a portion of the guidance data to a display device communicatively connectable to the data output interface. The storage unit includes instructions, when executed by the processing device, that cause the computing device to perform a method for assisting in instrument positioning according to any one of the embodiments disclosed herein.

[0062] According to embodiments, the computing device is a stand-alone computer communicatively connected to the intraoperative imaging device. Alternatively or additionally, the computing device is a remote computer (e.g., a cloud-based computer) communicatively connected to the intraoperative imaging device using a communications network, in particular at least in part using a mobile communications network. Alternatively or additionally, the computing device is integrated into the intraoperative imaging device or the display device.

[0063] Another object of the present invention is to provide a system for positioning an instrument relative to a specific body part of a patient that can reconstruct anatomical 3D geometry and generate a visual representation of the position of the instrument relative to the specific body part of the patient using only intraoperative 2D imaging data, i.e., without requiring navigation hardware to be installed in the operating room and without requiring a preoperative planning registration process.

[0064] The above-identified objectives are further achieved by a system including a computing device, an intraoperative imaging device, and a display device according to any one of the embodiments disclosed herein, the system being configured to perform a method according to any one of the embodiments disclosed herein. The intraoperative imaging device is communicatively connected to the computing device and positioned near a patient, allowing the intraoperative imaging device to capture intraoperative 2D imaging data of the patient, whereby two or more of the intraoperative 2D images capture a specific body part of the patient from two or more different viewpoints relative to the specific body part. One or more of the same multiple intraoperative 2D images capturing the specific body part also capture at least a portion of an instrument from at least one viewpoint. In the case of radiation-based images as intraoperative images, the intraoperative imaging device includes a C-arm intraoperative imaging device based on X-ray technology. The C-arm intraoperative imaging device includes a generator (X-ray source) and an image intensifier or flat-panel detector. A C-shaped connecting element allows movement horizontally, vertically, and / or about a swivel axis, so that 2D X-ray images of the patient can be generated from various viewpoints around the patient. The generator emits x-rays that penetrate the patient's body. An image intensifier or detector converts the x-rays into a visible image that is transmitted to a computing device. The display device can be a computer screen, an augmented reality headset, or any device configured to display the guided data.

[0065] Another object of the present invention is to provide a computer program product for positioning of an instrument relative to a specific body part of a patient that can reconstruct anatomical 3D geometry and generate a visual representation of the position of the instrument relative to the specific body part of the patient using only intraoperative 2D imaging data, i.e., without requiring navigation hardware to be installed in the operating room and without requiring a preoperative planning registration process.

[0066] The above-identified objects are addressed by a computer program product comprising instructions that, when executed by a processing unit of a computing device, cause the computing device to perform a method according to any one of the embodiments disclosed herein.

[0067] According to an embodiment, the instructions (included in the computer program product) include an artificial intelligence algorithm corresponding to a particular body part of the patient, the artificial intelligence algorithm being trained using a number of annotated imaging datasets capturing body parts corresponding to the particular body part of the patient, and the annotations including data identifying and / or describing characteristics of the body part.

[0068] According to an embodiment, the instructions (included in the computer program product) include instructions for controlling an intraoperative imaging device to capture intraoperative 2D imaging data including intraoperative 2D images, wherein the plurality of intraoperative 2D images capture a particular body part of the patient from a plurality of different viewpoints relative to the particular body part of the patient, and one or more of the plurality of intraoperative 2D images capture at least a portion of an instrument from at least one viewpoint.

[0069] According to an embodiment, the instructions (included in the computer program product) include instructions for controlling a display device to display at least a portion of the guidance data including a visual representation of the estimated current position of the instrument, a visual representation of the reconstructed anatomical 3D shape, and / or a visual representation of the defined position of the instrument, etc.

[0070] It is to be understood that both the foregoing general description and the following detailed description present embodiments and are intended to provide an overview or framework for understanding the nature and character of the present disclosure. The accompanying drawings are included to provide a further understanding, and are incorporated in and constitute a part of this specification. The drawings illustrate various embodiments, and together with the description, serve to explain the principles and operation of the disclosed concepts.

[0071] The term "particular" is used herein to refer to embodiments of the invention without any indication of preference, and without any indication that the features introduced as particular are essential to all embodiments of the invention.

[0072] The present disclosure will now be explained in more detail, by way of example, with reference to the drawings. [Brief explanation of the drawings]

[0073] [Figure 1] 1 is a flow chart illustrating steps of a method for processing anatomical imaging data in accordance with the present invention, including generating a training data set for use in training an artificial intelligence algorithm. [Figure 2] 1 is a flowchart illustrating steps in a process for determining a camera matrix. [Figure 3] FIG. 1 illustrates a data collection and processing pipeline for a specific application of the method for processing anatomical imaging data of the present invention to a fresh-frozen ex vivo whole stem specimen. [Figure 4] 10A-10C show exemplary 2D images during the process of restoring a calibration device from the 2D image (for the specific application of FIG. 3). [Figure 5]1 is a flowchart showing steps of a method for processing anatomical imaging data by applying a transfer learning process, including generating a training dataset, pre-training an artificial intelligence algorithm using the pre-training dataset, training the pre-trained artificial intelligence algorithm with the training dataset, and processing the anatomical imaging data using the trained artificial intelligence algorithm. [Figure 6] FIG. 1 illustrates a particular embodiment of retraining a pre-trained artificial intelligence algorithm by applying a transfer learning process. [Figure 7] 10 is a flowchart showing steps of a method for processing anatomical imaging data by applying a style transfer process, including generating a training dataset, pre-training an artificial intelligence algorithm using the pre-training dataset, training a style transfer algorithm using the training dataset, pre-conditioning the anatomical imaging data using the style transfer algorithm, and processing the pre-conditioned anatomical imaging data using the pre-trained artificial intelligence algorithm. [Figure 8] 1 is a flow chart illustrating steps for applying a style transfer algorithm to anatomical imaging data. [Figure 9] FIG. 1 is a diagram of a first embodiment of a style transfer algorithm based on deep convolutional generative adversarial networks (DCGANs). [Figure 10] FIG. 1 is a diagram of a second embodiment of a style transfer algorithm based on cycle-consistent generative adversarial networks (CycleGAN). [Figure 11] 1 is a highly schematic perspective view of a system for assisting in instrument positioning installed in an operating room, according to one embodiment of the present invention; [Figure 12] 1 is a flowchart illustrating steps of a method for assisting in instrument positioning, according to one embodiment of the present invention. [Figure 13]1 is a flowchart illustrating steps for reconstructing an anatomical 3D shape based on intraoperative 2D imaging data and data indicative of a camera matrix corresponding to a plurality of intraoperative 2D images, according to an embodiment of the present invention. [Figure 14] FIG. 1 is a schematic diagram of segmenting intraoperative 2D imaging data to identify specific body parts of a patient. [Figure 15] FIG. 10 is a schematic diagram of a further embodiment of segmenting intraoperative 2D imaging data, including identifying a region of interest, followed by semantic segmentation of the region of interest to identify a particular body part of the patient within the region of interest. [Figure 16] Schematic illustration of anatomical 3D shape reconstruction based on segmented intraoperative 2D imaging data using artificial intelligence algorithms corresponding to specific body parts. [Figure 17] 10 is a flowchart showing steps of a further embodiment of reconstructing an anatomical 3D shape in multiple stages. [Figure 18] FIG. 1 is a schematic diagram of determining a defined position of an instrument. [Figure 19] Figure 19A shows an example of a visual representation of an estimated current position of an instrument and a visual representation of a prescribed position of an instrument superimposed on a visual representation of a reconstructed anatomical 3D shape, Figure 19B shows an example of a visual representation of an estimated current position of an instrument superimposed on a 2D image of intraoperative 2D imaging data ID, and Figure 19C shows an example of a visual representation of an estimated current position of an instrument, a visual representation of a prescribed position of an instrument, and a visual representation of an ideal thread trajectory of a surgical implant on a visual representation of a reconstructed anatomical 3D shape. DETAILED DESCRIPTION OF THE INVENTION

[0074] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings, in which some, but not all, features are shown. Indeed, the embodiments disclosed herein may be embodied in many different forms and should not be construed as limitations to the embodiments set forth herein; rather, such embodiments are provided so that this disclosure will satisfy applicable legal requirements. Wherever possible, like reference numbers are used to refer to like components or parts.

[0075] The steps of a method for processing anatomical imaging data according to the present invention, including generating a training data set for use in training an artificial intelligence algorithm, are shown in FIG.

[0076] In a first step S10, 3D imaging data (referred to as acquired 3D imaging data) of particular body parts of anatomical specimens among a plurality of anatomical specimens is acquired. In a first sub-step S12 of step S10, 3D imaging data, in particular computed tomography (CT) scans, of the plurality of anatomical specimens are acquired using a 3D imaging device. In an optional sub-step S14 of step S10, individual body parts, such as vertebrae, are segmented within the 3D volume.

[0077] In step S30, 2D images are acquired after or simultaneously with step S10. In sub-step S32, 2D images of a particular body part of an anatomical specimen are acquired from a plurality of different viewpoints of the particular body part (referred to as acquired 2D images).

[0078] In step S40, multiple camera matrices P corresponding to the acquired 2D images are determined, whereby the camera matrix P indicates the viewpoint of the 2D image for at least a specific body part. Details of determining the multiple camera matrices P will be described below with reference to FIG. 2. Finally, in step S50, a training data set is generated. In a first sub-step S52 of step S50, multiple camera matrices P corresponding to the acquired 2D images are determined by the computing device 10, and as a further sub-step S54 of step S50 (generation of training data set), a composite 2D image is generated by the computing device by projecting the acquired 3D imaging data according to the multiple camera matrices P. Thus, one composite 2D image is generated for each camera matrix P, i.e., for each acquired 2D image viewpoint. As a further sub-step S56 of step S50, the acquired 2D images according to the multiple camera matrices P, in particular the multiple viewpoints, are associated with one of the composite 2D images as a training data pair. In other words, each 2D image is associated with a composite 2D image generated according to a camera matrix P that corresponds to the viewpoint of the acquired 2D image.

[0079] In optional step S20, 3D imaging data is acquired with calibration devices attached to the anatomical specimen to enable precise determination of a plurality of camera matrices P corresponding to the acquired 2D images (see step S30) and to enable projection of the acquired 3D imaging data according to the plurality of camera matrices P. In sub-step S22 of step S20, n calibration devices are provided to the anatomical specimen before acquiring 3D imaging data and 2D images of specific body parts of the anatomical specimen. In sub-step S24, 3D imaging data of specific body parts of the anatomical specimens (referred to as acquired 3D imaging data) of multiple anatomical specimens are acquired with the attached calibration devices. In sub-step S26, to compensate for resulting registration errors, computing device 10 performs 3D-3D registration of the 3D imaging data acquired before providing the anatomical specimen with the calibration devices and the 3D imaging data acquired after providing the anatomical specimen with the calibration devices. After 3D-3D registration of the 3D imaging data acquired before providing the calibration device to the anatomical specimen and the 3D imaging data acquired after providing the calibration device to the anatomical specimen, a composite 2D image is generated (at step S54) by projecting the 3D imaging data acquired before providing the calibration device to the anatomical specimen according to the multiple camera matrices P. In this way, the calibration device is not present in the composite 2D image (it is only present in the anatomical specimen).

[0080] In optional sub-step S34 of step S30, in the case where the 2D image is acquired with an attached calibration device, in order to remove the calibration device captured in the 2D imaging data, according to an embodiment, the projection of the calibration device in the 2D image is restored / removed from the 2D image.

[0081] 2 shows the substep S40 of determining the camera matrix P. In the first substep S41, the 2D coordinates of the calibration device are detected with respect to the acquired 2D image (referred to as the detected 2D coordinates), for example, using a circular Hough transform algorithm applied to the thresholded 2D image. In substep S42, the 3D coordinates of the calibration device are determined based on the 3D imaging data (referred to as the detected 3D coordinates), for example, by initial thresholding. Subsequently, in a sequence of iterative substeps S43 to S46, all possible correspondences between the 2D and 3D coordinates (referred to as the 2D-3D coordinate correspondences) and the respective camera matrix P are determined, and for each possible correspondence between the 2D and 3D coordinates, the 3D coordinates are projected and the reprojection error is calculated. In the first iterative substep S43, the (next) possible 2D-3D coordinate correspondence of the calibration device is selected. Subsequently, in substep S44, the camera matrix P corresponding to the acquired 2D image is determined based on the currently selected 2D-3D coordinate correspondence, for example, using a direct linear transformation algorithm. Following the determination of the camera matrix P for the currently selected 2D-3D coordinate correspondence, in substep S45, the 3D imaging data (captured after providing the calibration device) is projected according to the multiple camera matrices P. The projected 3D imaging data is superimposed with the corresponding acquired 2D image, thereby calculating the so-called reprojection error—substep S46. This sequence of substeps S43 to S46 is repeated for all possible correspondences between 2D and 3D coordinates. The number of possible correspondences between 2D and 3D coordinates is obtained from the number n of calibration devices captured by both 2D and 3D images according to a combination formula:

[0082]

number

[0083] After calculating the reprojection error for all possible correspondences between 2D and 3D coordinates, the 2D-3D coordinate correspondence with the lowest reprojection error is selected as the correct one—substep S47. This 2D-3D coordinate correspondence is then used to determine the camera matrix P for each of the acquired 2D images.

[0084] According to an embodiment, in order to enable a more precise determination of the camera matrix P, a relatively large number of calibration devices are provided, and in order to limit the number of all possible correspondences between 2D and 3D coordinates, only a first subset n' of the calibration devices (called reference calibration devices) is considered in the sequence of substeps S43 to S47. In this case, the number n' defining the number of all possible correspondences is the number of reference calibration devices.

[0085] Thereafter, having determined the 2D-3D coordinate correspondences of the reference calibration device, in optional sub-step S48, all calibration devices (not just the reference calibration device) are projected according to the selected multiple camera matrices P, and estimated 2D coordinates of all calibration devices are determined from these projections. Based on a comparison of these estimated 2D coordinates with the detected 2D coordinates of sub-step S41, 2D-3D coordinate correspondences are determined for all calibration devices, which allows for recalculation (in sub-step S49) of the camera matrix P utilizing all calibration devices. Thus, an optimal balance can be achieved between fitting a large number of calibration devices to enable accurate camera matrix P determination and the computational power required to determine the re-projection error of each camera matrix P using all possible 2D-3D correspondences.

[0086] The data collection and processing pipeline for the specific application of fresh-frozen ex vivo whole trunk specimens, previously described in Figures 1, 2, and 4, is shown in Figure 3. The process begins in step S12 by acquiring CT scans (e.g., slice thickness: 0.75 mm, in-plane resolution: 0.5 mm x 0.5 mm) of multiple fresh-frozen ex vivo whole trunk specimens, based on which, in step S14, segmentation of individual body parts, such as vertebrae, is performed within the 3D volume.

[0087] Thereafter, in step S22, stainless steel spherical fiducials (a specific embodiment of the calibration device) are placed into the frozen soft tissue by first inserting a K-wire (3 mm diameter) indicating the intended drilling trajectory, visually verifying the intended trajectory using fluoroscopic guidance, and, if confirmed, performing subsequent drilling to create narrow tunnels within the frozen tissue for fiducial placement: seven fiducials with a 2.5 mm radius and seven fiducials with a 1.5 mm radius. After the fiducial insertion step, any residual tunnels remaining within the frozen soft tissue can be sealed using low-viscosity superglue. When placing the fiducials, it should be ensured that the bony structures remain intact and that the fiducials are inserted within an imaginary cylinder that encapsulates the spinal column and covers the area from L1 to L5. This is to ensure that the fiducials have sufficient separation and are visible at clinically relevant imaging angles (AP, lateral, and oblique).

[0088] Thereafter, in step S24, a fiducial placement CT image is acquired of the anatomical specimen explicitly showing the relative placement of the fiducials to the tissue. Then, in step S26, a 3D-3D registration between the initial CT scan (fixed image) and the fiducial placement CT scan is performed, followed by extraction of the 3D fiducial coordinates Xi∈R from this CT image through an initial thresholding of metal objects in step S41. 3 A list containing ,i∈[1,2,...,14] is extracted and later 3D connected component analysis is performed.

[0089] In a further step S32, X-ray images of the anatomical specimen are acquired, starting with the anterior-posterior (AP) view as the base view, using, for example, a clinical-grade mobile C-arm device. Starting from the AP view, the specimen is imaged in the transverse plane by rotating the C-arm gantry in 3° increments until an orbital angle of ±102° is achieved in both directions. Furthermore, starting from the AP view, tilting of the C-arm is used in 3° increments to image the specimen in the sagittal plane until a tilt angle of ±25° is achieved in both directions, resulting in an oblique X-ray. Similarly, starting from the lateral view, tilting is used in 5° increments in the sagittal plane until a tilt of ±15° is achieved. During the image acquisition phase, the C-arm gantry is positioned so that the center of the X-ray is at L3, capturing as much of the lumbar region as possible.

[0090] Once the X-rays are acquired, a calibration process - step S43 - is used to calibrate the X-rays using the coordinates of the projected fiducial markers in 2D and the corresponding 3D coordinates (details of the calibration process are described below with reference to Figure 4).

[0091] The computed camera matrix P is decomposed into extrinsic and intrinsic parameters by:

[0092]

number

[0093] where K is a matrix containing the intrinsic camera parameters, R is the rotation matrix, and X o are the focal coordinates and qr is used to decompose the matrix into an orthogonal matrix q and an upper triangular matrix r.

[0094] Thereafter, in sub-step S52 of step S50, using the recovered intrinsic and extrinsic camera parameters, a DRR image I is generated using the recovered intrinsic and extrinsic camera parameters using:

[0095]

number

[0096] where I(p) is the intensity of the DRR at image point p, and T = [R|X o]:R 3 →R 3 is the transformation between the object plane and the image plane, and L(p,s) is the ray starting from the X-ray source and intersecting the image plane at point p parameterized by s. Following this approach, the DRR image is projected from the same viewpoint as the acquired X-ray image to obtain a pair of synthetic-real X-ray data.

[0097] The entire process described above (i.e., pre-operative CT, fiducial placement, fiducial post-CT, calibration, and downstream DRR generation) is performed on all x-ray images collected from the anatomical specimen. A 3D segmentation mask is used to generate a 2D mask on the DRR image, and the 2D mask is used to localize the synthetic-real x-ray image pair so that only a single spinal level is present on a given image.

[0098] As shown in FIG. 4, the process for determining the camera matrix P for the particular application of FIG. 3 is as follows. Reference detection - Substep S41: The projected criterion xj∈R 2 The 2D coordinates of,j∈[1,2,...,14] are first found using a circular Hough transform algorithm applied to the thresholded image. This can be displayed as a suggestion to the user (Fig. 4a), who can accept, reject, or modify the detected points using keyboard and mouse input.

[0099] 2D-3D coordinate correspondence - calculations in substeps S43 to S47: Up to this stage, 2D and 3D reference coordinates have been extracted, but the coordinate x∈R 2 and x∈R 3The correspondence between the two lists of X and R was not addressed. For this purpose, the bead with the larger radius, whose corresponding 2D coordinates are desired in a first step, and its 3D coordinates as the reference standard are considered. Given a given 2D-3D correspondence, a direct linear transformation (DLT) algorithm can be used to calculate the camera matrix P (including intrinsic and extrinsic imaging parameters) and the associated reprojection error. This reprojection error is then used as a cost function within a random sample consensus (RANSAC) framework, whereby the calculation of the P matrix and the resulting reprojection error is iterated for all possible 2D-3D correspondences of the reference standard. The particular correspondence that results in the smallest reprojection error is considered correct and is used to calculate the camera matrix P, which is an initial estimate of the camera matrix P (calculated using only the reference standard) - substep S47. This camera matrix P is then calculated in the 3D space X∈R. 3 is used to project all reference points from R to the 2D image space by x = PX, where x∈R 2 are the estimated 2D coordinates of the projected fiducials (Fig. 4b).

[0100] Reference 2D-3D coordinate correspondence correction - Substep S48: Estimated reference projection

[0101]

number

[0102] By comparing x with the detected fiducial projections x, the 2D-3D coordinate correspondences for all of the fiducials are recovered, allowing the camera matrix P to be (re)calculated—substep S49—using all of the fiducial markers (FIG. 4c).

[0103] Reference Repair - Step S34: The projection of the fiducial markers is then restored to the acquired X-ray (Fig. 4d). FIG. 5 shows a flowchart illustrating steps of a method for applying a transfer learning process to process anatomical imaging data, including generating a training dataset, pre-training an artificial intelligence algorithm using the pre-training dataset, training the pre-trained artificial intelligence algorithm with training data pairs from the training dataset, and processing the anatomical imaging data using the trained artificial intelligence algorithm.

[0104] In step S60, an artificial intelligence algorithm is pre-trained using a pre-training dataset. In a first sub-step S62, a pre-training dataset is obtained. To overcome limitations in the availability and / or accuracy of annotated imaging datasets capturing a body part corresponding to a specific body part, the pre-training dataset includes a large number of imaging datasets generated from 3D imaging data, particularly computed tomography (CT) scans, capturing the specific body part. In particular, given an input pre-operative CT scan, synthetic 2D images, such as fluoroscopy shots (i.e., DRRs), are generated with various camera matrices P, particularly from various viewpoints. For example, using this method, up to several hundred "synthetic" 2D images (capturing a specific body part) can be generated from a single CT scan. In a subsequent sub-step S64 of step S60, an artificial intelligence algorithm is pre-trained using the pre-training dataset.

[0105] Thereafter, in step S70, the pre-trained artificial intelligence algorithm is (re)trained using a training dataset generated according to one of the embodiments disclosed herein (described with reference to FIGS. 1 to 4). Training (also referred to as retraining) the pre-trained artificial intelligence algorithm includes attaching a first subset of network parameters, such as artificial neural network weights and biases, for the network components of the pre-trained artificial intelligence algorithm and adjusting a second subset of the network parameters using the training data pairs of the training dataset as inputs. In a specific embodiment for reconstructing the 3D anatomical shape of a specific body part of an anatomical object, retraining the pre-trained artificial intelligence algorithm includes attaching network parameters, such as artificial neural network weights and biases, for the network components of the pre-trained artificial intelligence algorithm, except for the 2D feature extractor module, and adjusting the network parameters of the 2D feature extractor module using the training data pairs of the training dataset as inputs.

[0106] As the next major step S80, a 2D image of a particular body part of an anatomical object, such as a patient undergoing surgery or medical examination, is acquired. Thereafter, in step S100, the acquired 2D images are processed using an artificial intelligence algorithm that has first been pre-trained using the pre-training data set and then (re-)trained using the training data set.

[0107] In a first (alternative or additional) sub-step S102 of step S100, processing the acquired 2D image of the particular body part of the anatomical object using an artificial intelligence algorithm includes reconstructing the anatomical 3D shape of the particular body part of the anatomical object based on the acquired 2D image of the particular body part and a camera matrix P, in particular data indicating a viewpoint corresponding to the acquired 2D image.

[0108] As a second (alternative or additional) sub-step S104, processing the acquired 2D image of the particular body part of the anatomical object using an artificial intelligence algorithm includes anatomically segmenting the acquired 2D image to identify one or more anatomical features of the particular body part.

[0109] 6 illustrates a specific embodiment of retraining a pre-trained artificial intelligence algorithm. Given that the artificial intelligence algorithm was pre-trained (and tested) using synthetic 2D images, the goal of retraining is to use the data pairs (of an acquired 2D image and a corresponding synthetic 2D image of a training dataset generated according to one of the embodiments disclosed herein) along with the recovered camera matrix P (i.e., the extrinsic and intrinsic imaging parameters) to retrain the pre-trained artificial intelligence algorithm in a transfer learning manner.

[0110] As shown in FIG. 6, the pre-trained artificial intelligence algorithm accepts as its input a localized 2D image along with the corresponding imaging parameters. As the first step in the re-training process, a 2D feature map is extracted using a UNet model, which is then passed to a differentiable backprojection module that uses the input imaging parameters (i.e., the camera matrix P) to create a 3D feature grid. The 3D feature grid is then averaged and fed into an improved UNet model, which outputs a reconstructed 3D shape. During the transfer learning process, the pre-trained artificial intelligence algorithm is retrained by attaching network parameters (weights and biases) for all of the network components except for the network parameters of the 2D feature extractor module. The trainable network parameters for the 2D feature extractor module are adjusted during the transfer learning process using the collected training data pairs as input.

[0111] FIG. 7 shows a flowchart illustrating steps of a method for processing anatomical imaging data by applying a style transfer process, including generating a training dataset, pre-training an artificial intelligence algorithm using the pre-training dataset, training a style transfer algorithm using the training dataset, pre-conditioning anatomical imaging data using the style transfer algorithm by applying style transfer to acquired 2D images, and processing the pre-conditioned anatomical imaging data using the pre-trained artificial intelligence algorithm.

[0112] As an alternative to the transfer learning approach shown in FIGS. 5 and 6, this approach according to the embodiment shown in FIGS. 7 and 8 can be used. In this approach, providing a domain adaptation framework is addressed by a style transfer process for acquired (real) 2D images using an artificial intelligence algorithm pre-trained on synthetic images. Similar to the transfer learning description shown in FIG. 5, step S50 of generating a training dataset and step S60 of pre-training an artificial intelligence algorithm using the pre-training dataset are performed. Then, instead of performing transfer learning as step S70 in FIG. 5, in step S65, a style transfer learning algorithm is trained using the training dataset. Similar to transfer learning, in step S70, a 2D image of an anatomical object is captured. Unlike the transfer learning shown in FIG. 5, the trained transfer learning algorithm is then applied to the acquired (or captured) 2D image in step S90. Similar to transfer learning, in step S100, an anatomical 2D image is processed, but in this case, compared to FIG. 5, it is processed together with the captured 2D image processed by the style transfer algorithm. In other words, instead of performing transfer learning (step S70), a style transfer algorithm is trained (S65) and the style transfer algorithm is applied to the acquired 2D images (S90). As a result, processing the anatomical 2D images (S100) is performed on the preconditioned / augmented 2D images.

[0113] In particular, in step S65, a style transfer algorithm is trained using the training data set so that the style transfer algorithm is capable of extracting texture information from the synthetic 2D image and transferring the texture information onto the acquired 2D image while preserving the underlying semantic content of the acquired 2D image.

[0114] Once the style transfer algorithm has been trained, in step S90, the style transfer algorithm is applied to the acquired 2D image (referred to as a pre-conditioned acquired 2D image) prior to processing the acquired 2D image (using an artificial intelligence algorithm).

[0115] Step S90 of applying the style transfer algorithm is shown in more detail in Figure 8. In sub-step S92 of the style transfer process, texture information is extracted from the synthetic 2D images of the training data set. In sub-step S94, pre-conditioned acquired 2D images of the anatomical object are synthesized by transferring the texture information onto the acquired 2D images while preserving the underlying semantic content of the acquired 2D images.

[0116] Known methods of style transfer for intraoperative 2D imaging data generally address the transfer from synthetic 2D imaging (i.e., DRR) to acquired 2D imaging (i.e., X-ray domain) in the hope of achieving more realistic synthetic 2D images used to train artificial intelligence algorithms. On the other hand, the main idea of ​​the style transfer approach of the present invention is to apply the style transfer concept in the reverse direction to transfer acquired (i.e., X-ray) 2D images into the synthetic 2D imaging (i.e., DRR) domain for the following reasons.

[0117] Even under controlled imaging conditions, acquired (i.e., X-ray) 2D images have high variability in image and radiometric characteristics depending on factors such as imaging technology, detector type (intensifier vs. flat panel), device settings, patient body mass index (BMI), imaging field of view, and backscatter. This makes the task of transferring synthetic 2D imaging to acquired 2D imaging (i.e., DRR to X-ray) inherently challenging, given the high heterogeneity of the target domain itself (i.e., X-ray domain) in this setup. In contrast, a fixed threshold is applied to the underlying acquired 3D image (i.e., CT scan) to create the synthetic (i.e., DRR) 2D image in accordance with the present invention. The effect is that all CT voxels with Hounsfield values ​​below the threshold are ignored during the DRR generation stage. This threshold is empirically selected to project only bone structures onto the synthetic (i.e., DRR) 2D image, resulting in a homogeneous target domain.

[0118] Similar to other artificial intelligence-based methods that use synthetic (i.e., DRR) 2D image data for training purposes, the pre-trained artificial intelligence algorithm of the present invention has already been trained using a large number of annotated synthetic (i.e., DRR) 2D images. Thus, the acquired (i.e., X-ray) 2D images can be first converted into the synthetic (i.e., DRR) domain by sending the acquired (i.e., X-ray) to a synthetic (i.e., DRR) style transfer algorithm, and then the pre-trained artificial intelligence algorithm can be used.

[0119] Using the training dataset (generated according to the methods of the present invention), four different types of style transfer algorithms can be applied, according to various embodiments. Encoder-Decoder Deep Convolutional Network (EDDC): This style transfer algorithm can be implemented using an hourglass encoder-decoder architecture. To train the EDDC network, a training dataset generated according to any of the embodiments of the present invention is used, which includes paired synthetic-real 2D images localized by considering acquired (i.e., X-ray) 2D images as network inputs and corresponding localized synthetic (i.e., DRR) images as network targets.

[0120] Neural Style Transfer (NST): This algorithm operates by separating style and content from input images using a deep convolutional network as a feature extractor. To generate an output image, content information from the acquired (i.e., X-ray) image is extracted and combined with style information abstracted from the synthetic (i.e., DRR) image. At the heart of this algorithm are two specialized loss functions: one tasked with estimating the content similarity between the generated image and the acquired input (i.e., X-ray) image (i.e., content loss), and the other serving to estimate the style similarity between the generated image and the input synthetic (i.e., DRR) image. To train NST, a localized paired synthetic-real training dataset generated according to any of the embodiments of the present invention is used.

[0121] Deep Convolutional Generative Adversarial Network (DCGAN): Generative adversarial networks are widely used for image synthesis tasks, where a GAN model is trained to generate new images based on chaotic noise inputs. According to the present invention, the GAN-based image synthesis task is transformed into a style transfer task by exchanging the chaotic noise inputs with real / acquired (i.e., X-ray) 2D images, as shown in Figure 9. This can be achieved thanks to a localized paired synthetic-real training dataset generated according to any of the embodiments of the present invention.

[0122] Cycle-Consistent Generative Adversarial Network (CycleGAN): In cases where only unpaired training data is available for style transfer, a CycleGAN network is preferred, including mechanisms for ensuring reversible image-to-image transfer and preventing mode collapse. To address this use case, the CycleGAN network architecture shown in FIG. 10 may be used, including two generator and two discriminator networks. If unpaired training data is sufficient to train such a CycleGAN network, a localized paired synthetic-real training dataset (generated according to any of the embodiments of the present invention) and an additional, larger synthetic (i.e., DRR) 2D imaging data dataset may be used. This may be performed to identify whether unpaired synthetic-real data alone is sufficient for the desired style transfer task.

[0123] 11 to 19, specific practical applications of the methods for generating training data for training an artificial intelligence algorithm that assists in positioning an instrument, such as a surgical instrument, relative to a particular body part of a patient (reconstructing the anatomical 3D shape (AS) of the particular body part based on intraoperative 2D imaging data (ID) and data indicating viewpoints corresponding to multiple intraoperative 2D images) are described.

[0124] FIG. 11 shows a highly schematic perspective view of a system 1 for assisting in the positioning of an instrument 5, installed in an operating room, with a patient 200 reclining on an operating table 2. As shown, the system 1 includes a computing device 10, an intraoperative imaging device 20, and a display device 30. The system 1 is shown based on an embodiment utilizing radiation-based images as intraoperative images. Accordingly, the intraoperative imaging device 20 includes a C-arm intraoperative imaging device 20 that captures intraoperative images using X-ray technology. The C-arm intraoperative imaging device 20 includes a generator (X-ray source) 22. A C-shaped connecting element (C-arm) 24 enables movement horizontally, vertically, and / or about a swivel axis, so that 2D X-ray images of the patient 200 can be generated from various viewpoints around the patient. The generator 22 emits X-rays that penetrate the patient's body 200. A detector 26 converts the X-rays into imaging data ID, which is transmitted to the computing device 10.

[0125] The intraoperative imaging device 20 is communicatively connected to the computing device 10 and positioned near the patient 200 enabling the intraoperative imaging device 20 to capture intraoperative 2D imaging data ID of the patient 200, whereby two or more of the intraoperative 2D images capture a particular body part 202 of the patient 200 from two or more different perspectives of the particular body part 202 of the patient 200. One or more of the same plurality of intraoperative 2D images capturing the particular body part 202 also captures at least a portion of the instrument 5 from at least one perspective.

[0126] In the illustrated embodiment, the display device 30 comprises a series of computer screens 32 communicatively connected to the computing device 10 and configured to display the guidance data GD.

[0127] Referring now to the flowchart of FIG. 12, the steps of a computer-implemented method for assisting in the positioning of an instrument 5 relative to a particular body part 202 of a patient 200 will be described. As shown in FIG. 12, the method includes the following main steps:

[0128] Step S110: Capturing intraoperative 2D imaging data; Step S120: Receiving intraoperative 2D imaging data; Step S130: Reconstructing an anatomical 3D shape using the intraoperative 2D imaging data and using an artificial intelligence based algorithm corresponding to the particular body part 202; Step S140: Estimating the current position 5c of the instrument based on the intraoperative 2D imaging data ID; Step S150: Identifying a defined position 5p of the instrument; Step S160: Reconstructing positioning guidance data GD including a visual representation of the estimated current position 5c of the instrument 5 relative to the anatomical 3D shape AS of the particular body part 202; and Step S170: Using the display device 30 to output the positioning guidance data GD.

[0129] Steps that are unique to a particular embodiment are indicated in the figures with dashed lines. In step S110, intraoperative 2D imaging data ID is captured by an intraoperative imaging device 20 positioned near the patient 200. The intraoperative 2D imaging data ID includes data indicating a viewpoint corresponding to the plurality of intraoperative 2D images, which identifies the position and / or orientation of the intraoperative imaging device 20 capturing the plurality of intraoperative 2D images, such as the position in an x, y, and z Cartesian coordinate system of the intraoperative imaging device 20 relative to a particular body part 202 of the patient 200, and / or its orientation as roll, pitch, and yaw.

[0130] According to a first embodiment, data indicating viewpoints corresponding to a plurality of intraoperative 2D images are stored in a data store included in or communicatively connected to the computing device 10. The viewpoints stored in the data store are determined by tracking the C-arm 24 and estimating imaging parameters at the time of exposure, through which intraoperatively acquired 2D images can be assigned respective intrinsic and extrinsic imaging parameters that substantially define the viewpoints from which the intraoperative 2D images were generated. Optionally, tracking the C-arm 24 is preceded by a calibration process, i.e., pre-calibration, in which the C-arm 24 is manipulated in a specific manner to cover an intended range of motion. During this pre-calibration phase, mathematical relationships between tracking observations and imaging parameters are established at specific posture intervals, and the mathematical relationships are later used to derive interpolation functions that can generate intraoperative imaging parameters based on the tracking data.

[0131] Alternatively or additionally, a viewpoint corresponding to the intraoperative 2D imaging data ID is estimated by the computing device 10 based on the intraoperative 2D imaging data ID. In one embodiment, a calibration algorithm extracts the viewpoint of the intraoperative 2D image by placing a precisely fabricated calibration object (i.e., phantom) containing distinct features (e.g., radiopaque features) with known geometry within the imaging field of view, and estimates imaging parameters based on the projection of such features.

[0132] Alternatively or additionally, estimating the viewpoint corresponding to the intraoperative 2D imaging data ID is performed using an artificial intelligence algorithm trained using multiple imaging datasets with known viewpoints. To overcome limitations in the availability and / or accuracy of imaging datasets with known viewpoints, multiple imaging datasets containing intraoperative 2D images from known viewpoints are generated from 3D imaging data, particularly computed tomography (CT) scans. For example, simulated intraoperative fluoroscopy shots (i.e., digitally reconstructed radiographs, DRRs) generated based on preoperative CT scans along with corresponding posture parameters are used to train a convolutional neural network (CNN) for the regression task. Using this preoperatively trained artificial intelligence algorithm, the intraoperative position of the intraoperative imaging device 20 can be estimated based solely on the intraoperative images, without the need for an external tracking device or calibration phantom.

[0133] In a subsequent step S120, the intraoperative 2D imaging data ID is received by the computing device 10 from the intraoperative imaging device 20 via its data input interface 14.

[0134] In a subsequent step S130, an anatomical 3D shape AS of the particular body part 202 is reconstructed by the computing device 10 based on the intraoperative 2D imaging data ID and data indicating viewpoints corresponding to the plurality of intraoperative 2D images using an artificial intelligence algorithm corresponding to the particular body part 202. A detailed description of step S130 of reconstructing the anatomical 3D shape AS is provided with reference to Figures 13, 14, 15, 16, and 17.

[0135] In step S140, the current position 5c of the instrument 5 relative to the anatomical 3D shape AS of the particular body part 202 is estimated based on the intraoperative 2D images of the imaging data ID capturing the instrument 5. The current position 5c of the instrument 5 is performed based on prior knowledge of the geometry of the instrument 5, which is described by an instrument geometric model. First, a projection of the instrument geometric model is compared to at least a portion of the instrument 5 captured by each 2D image of the intraoperative 2D imaging data ID. The instrument geometric model is projected onto one or more planes of the intraoperative 2D images of the intraoperative 2D imaging data ID capturing at least a portion of the instrument 5. The planes of the intraoperative 2D images of the intraoperative 2D imaging data ID are determined based on the viewpoint of each 2D image. Then, a position of the instrument geometric model is determined that generates a projection onto the plane of the intraoperative 2D images of the intraoperative 2D imaging data ID that (best) matches at least a portion of the instrument 5 captured by each 2D image of the intraoperative 2D imaging data ID.

[0136] According to the embodiments disclosed herein, at the initial stage of the method for assisting in instrument positioning, the anatomical 3D shape is reconstructed once, but estimation of the current position 5c of the instrument 5 is performed repeatedly at set intervals and / or triggered by certain events and / or triggered manually.

[0137] To improve the accuracy of estimating the position of the instrument 5, the instrument 5 is certified according to an instrument geometric model. The instrument geometric model is specifically designed to optimize the estimation of the instrument's position based on as few intraoperative 2D images as possible. In particular, to enable estimation of the instrument's orientation based on the intraoperative 2D images, the instrument is designed so that at least a portion of it is not perfectly rotationally symmetric about any of the axes of a Cartesian coordinate system. Alternatively or additionally, the instrument is designed with special markers to facilitate identification of the instrument based on the 2D intraoperative images.

[0138] In step S150, the defined position 5p of the instrument 5 relative to the anatomical 3D shape AS of the particular body part 202 is identified by the computing device 10, and a visual representation of the defined position 5p of the instrument 5 is overlaid on a visual representation of the estimated current position 5c of the instrument 5 to assist the surgeon in correctly positioning the instrument 5.

[0139] One embodiment for determining the defined position 5p of the instrument 5 will now be described with reference to FIG. FIG. 13 shows a flowchart illustrating steps for reconstructing an anatomical 3D shape AS based on intraoperative 2D imaging data ID and data indicating viewpoints corresponding to multiple intraoperative 2D images. As shown, reconstructing the anatomical 3D shape AS is performed in two stages: step S132—segmenting the intraoperative 2D imaging data ID to identify specific body parts 202 of the patient 200; and step S134—further using the segmented intraoperative 2D imaging data ID to reconstruct the anatomical 3D shape AS. Step S132—segmenting the intraoperative 2D imaging data ID to identify specific body parts 202 of the patient 200 using an artificial intelligence-based detection and segmentation model, as applied to segmenting intraoperative 2D images of the spine to identify individual vertebrae, is shown in FIG. 14. To train the artificial intelligence-based detection and segmentation model, given an input preoperative CT scan, synthetic X-rays (i.e., DRRs) are generated from various viewpoints around the patient 200. The CT scans used for this purpose may be collected through a public dataset containing CT scans and corresponding vertebral level annotations. For example, using this method, a training database of over 40,000 annotated intraoperative 2D images can be created from as few as 200 preoperative CT scans.

[0140] 15 shows a schematic diagram of a further embodiment of step S132 of segmenting intraoperative 2D imaging data ID using a two-stage approach including identifying a region of interest and semantically segmenting the region of interest to identify a specific body part 202 of the patient 200 within the region of interest. To segment the intraoperative imaging data ID, a region of interest including a specific body part 202 of the patient 200 is first identified within the intraoperative 2D imaging data ID using an artificial intelligence-based detection and segmentation model, such as a convolutional neural network-based detection and segmentation model. The region of interest is then semantically segmented using the artificial intelligence-based detection and segmentation model, thereby generating a segmented intraoperative 2D imaging data ID. Supervised learning is used to train the artificial intelligence-based detection and segmentation model used for the segmentation of step S132. First, a convolutional neural network (CNN)-based detection model is trained to identify individual body parts (vertebral levels in the illustrated example) on intraoperative 2D images by detecting the coordinates of bounding boxes that each contain a single body part (a single vertebra). The identified bounding boxes are then used as regions of interest for cropping the intraoperative 2D images. Furthermore, an end-to-end segmentation model is trained to semantically segment the projections of vertebrae within the regions of interest. During the inference phase, intraoperative X-ray images are fed into the segmentation model, which generates a semantic segmentation for each vertebral level (used for 3D reconstruction).

[0141] FIG. 16 illustrates a method for segmenting intraoperative 2D imaging data ID using an artificial intelligence algorithm corresponding to a particular body part 202 and a viewpoint P corresponding to multiple intraoperative 2D images. 1-n1 shows a schematic diagram of the reconstruction of the anatomical 3D shape AS of a particular body part 202 based on data representing the intraoperative 2D imaging data ID and the anatomical 3D shape AS of the particular body part 202. As shown, the segmented intraoperative 2D imaging data ID are back-projected into a 3D coordinate system to create an anatomical 3D shape for each body part 202 (in this case, a vertebra). The back-projection is performed from the viewpoint P of the intraoperative 2D image. 1-n , where each 2D image provides information about the body part 202 from that viewpoint. Thus, the anatomical 3D shape AS is constructed incrementally; the more intraoperative 2D images the intraoperative data ID contains, the more accurate the reconstructed anatomical 3D shape AS, as shown by the sequence of partial anatomical 3D shapes at the bottom of FIG. 16 .

[0142] In addition to steps S132 and S134 described with reference to FIG. 13, according to a further embodiment shown in the flowchart of FIG. 17, in another step S136, the anatomical 3D shape AS init The initial reconstruction of the anatomical 3D shape AS is further improved using unsegmented imaging data. Given potential errors in calibration and segmentation, the reconstructed initial anatomical 3D shape AS init To improve the quality of the anatomical 3D shape, a 3D shape refinement model is utilized. The 3D shape refinement model, specifically a convolutional neural network (CNN) architecture, has two input streams. The first input stream is the anatomical 3D shape AS init The second input stream includes a 2D segmentation of a particular body part 202 on the intraoperative 2D image of the intraoperative 2D imaging data ID. In this way, patient-specific geometric information carried in the original intraoperative 2D image is infused, thereby generating an improved anatomical 3D shape AS enh By reconstructing the initial reconstruction, a 3D shape refinement model is trained to complete the missing components of the initial reconstruction (assuming possible data loss in the initial reconstruction process due to missing projection views).

[0143] 18 , one embodiment for determining a prescribed position 5p of an instrument 5 will be described with reference to the use case of a surgical procedure to implant pedicle screws into the vertebrae of a patient 200. The prescribed position 5p of the instrument 5 is determined by an artificial intelligence-based optimization function based on the anatomical 3D shape AS of a particular body part 202 as well as data indicative of the surgical procedure.

[0144] Determining the prescribed position 5p of the instrument 5 based on the anatomical 3D shape AS is advantageous because it has the potential to improve surgical workflow by eliminating the need for preoperative scanning and the corresponding manual process, which can be costly and time-consuming. Supervised learning and reinforcement learning (RL) are used to train an artificial intelligence-based optimization function based on a clinical dataset containing ideal screw trajectories identified by experts. The prescribed position 5p of the instrument 5 is then determined based on the ideal screw trajectories IST, further based on prior knowledge of the geometry of the instrument 5.

[0145] 19A, 19B, and 19C show embodiments of positioning guidance data GD. Fig. 19A shows positioning guidance data GD including a visual representation of an estimated current position 5c of instrument 5 and a visual representation of a defined position 5p of instrument 5 superimposed on a visual representation of a reconstructed anatomical 3D shape AS.

[0146] FIG. 19B shows the positioning guidance data GD including a visual representation of the estimated current position 5c of the instrument 5 superimposed on the 2D image of the intraoperative 2D imaging data ID. FIG. 19C shows positioning guidance data GD including a visual representation of the estimated current position 5c of the instrument 5, a visual representation of the defined position 5p of the instrument 5, and a visual representation of the ideal screw trajectory of the surgical implant on a visual representation of the reconstructed anatomical 3D shape AS. [Explanation of symbols]

[0147] 1 System 2 Operating table 5. Equipment 5c Current location of the device 5p Specified position of the instrument 10. Computing Devices 12 Data output interface (of a computing device) 14 Data input interface (of a computing device) 16 Processing unit (of a computing device) 18 Storage unit (of a computing device) 20 Imaging Device 22 Generator (X-ray source) 24 C-shaped connecting element (C-arm) 26 detector 30 Display Devices 32 Computer Screen Arrangement 200 patients 202 (Patient) specific body part ID Intraoperative imaging data GD Positioning Guidance Data AS Anatomical 3D shape AS init Initial reconstruction of anatomical 3D shape AS enh Improved anatomical 3D shape IST Ideal pedicle screw trajectory P 1-n Viewpoint (of 2D intraoperative images)

Claims

1. 1. A computer-implemented method for processing anatomical imaging data, comprising: acquiring 3D imaging data of a particular body part of an anatomical specimen from the plurality of anatomical specimens; acquiring 2D images of the particular body part of the anatomical specimen from a plurality of different viewpoints relative to the particular body part; determining a plurality of camera matrices P corresponding to the acquired 2D images, whereby the camera matrices P indicate at least a viewpoint of the 2D images relative to the particular body part; generating a composite 2D image by projecting the acquired 3D imaging data according to the multiple camera matrices P; associating the acquired 2D image with one of the composite 2D images as a training data pair according to the plurality of camera matrices P; Including, The method further includes using the training data pairs to generate a training data set for use in training an artificial intelligence algorithm for processing anatomical imaging data. Computer-implemented methods.

2. 10. The computer-implemented method of claim 1, providing a calibration device to the anatomical specimen prior to acquiring the 3D imaging data and the 2D image of the particular body part of the anatomical specimen; determining the 2D coordinates of the calibration device in each of the 2D images; determining the 3D coordinates of the calibration device based on the 3D imaging data; determining a 2D-3D coordinate correspondence between the 2D coordinates of the calibration device in each of the 2D images and the 3D coordinates of the calibration device; further using the 2D-3D coordinate correspondences to determine the plurality of camera matrices P corresponding to the acquired 2D images; The computer-implemented method further comprising:

3. 3. The computer-implemented method of claim 2, further comprising: obtaining 3D imaging data capturing the particular body part of the anatomical specimen prior to providing the calibration device to the anatomical specimen; performing 3D-3D registration of the 3D imaging data acquired before providing the calibration device to the anatomical specimen and the 3D imaging data acquired after providing the calibration device to the anatomical specimen; generating the composite 2D image by projecting the 3D imaging data acquired prior to providing the calibration device to the anatomical specimen according to the plurality of camera matrices P; recovering a projection of the calibration device from the acquired 2D images prior to use in generating the training data set; The computer-implemented method further comprising:

4. 4. The computer-implemented method of claim 2 or 3, wherein determining the plurality of camera matrices P corresponding to the acquired 2D images comprises: detecting 2D coordinates of the calibration device on the acquired 2D image; detecting 3D coordinates of the calibration device on the acquired 3D imaging data; determining 2D-3D coordinate correspondence between the 2D and 3D coordinates of the calibration device; determining the camera matrix P using the 2D and 3D coordinates of the calibration device and the 2D-3D coordinate correspondences; 11. A computer-implemented method comprising:

5. 5. A computer-implemented method according to any one of claims 1 to 4, comprising: obtaining a pre-training dataset comprising a plurality of synthetic 2D images capturing the particular body part of the anatomical specimen; using the pre-training data set to pre-train an artificial intelligence algorithm for processing anatomical imaging data; The computer-implemented method further comprising:

6. 6. The computer-implemented method of claim 5, further comprising using the training data set to train a pre-trained artificial intelligence algorithm for processing anatomical imaging data.

7. 7. The computer-implemented method of claim 6, wherein training the pre-trained artificial intelligence algorithm comprises: appending a first subset of network parameters of the pre-trained artificial intelligence algorithm; adjusting a second subset of network parameters of the pre-trained artificial intelligence algorithm using the training data pairs of the training data set as inputs; 11. A computer-implemented method comprising:

8. 6. The computer-implemented method of claim 5, training a style transfer algorithm using the training dataset; applying a style transfer process to the acquired 2D image using the style transfer algorithm; The computer-implemented method further comprising:

9. 9. The computer-implemented method of claim 8, wherein the style transfer process comprises: extracting texture information from synthetic 2D images of the training dataset; transferring the texture information onto the acquired 2D images of the training data pairs while preserving the underlying semantic content of the acquired 2D images; 11. A computer-implemented method comprising:

10. 10. A computer-implemented method according to any one of claims 5 to 9, comprising: acquiring a 2D image of the particular body part of an anatomical object; processing the acquired 2D image of the particular body part of the anatomical object using the trained and / or pre-trained artificial intelligence algorithm; The computer-implemented method further comprising:

11. 11. The computer-implemented method of claim 10, wherein processing the acquired 2D images of the particular body part of the anatomical object using the artificial intelligence algorithm comprises reconstructing an anatomical 3D shape (AS) of the particular body part of the anatomical object based on the acquired 2D images of the particular body part and data indicative of a camera matrix (P) corresponding to the acquired 2D images.

12. 12. The computer-implemented method of claim 10 or 11, wherein processing the acquired 2D image of the particular body part of the anatomical object using the artificial intelligence algorithm comprises anatomically segmenting the acquired 2D image to identify one or more anatomical features of the particular body part.

13. 13. A computer-implemented method according to any one of claims 5 to 12, comprising the steps of assisting in positioning an instrument (5) relative to a particular body part (202) of a patient (200), the method comprising: a) receiving, by a computing device (10), intraoperative imaging data (ID) from an imaging device (20) positioned near the patient (200), the intraoperative imaging data (ID) including intraoperative 2D images, the plurality of intraoperative 2D images capturing the particular body part (202) of the patient (200) from a plurality of different viewpoints of the particular body part (202) of the patient (200), one or more of the plurality of intraoperative 2D images capturing at least a portion of the instrument (5) from at least one viewpoint; b) the computing device (10) reconstructs an anatomical 3D shape (AS) of the specific body part (202) using the trained artificial intelligence algorithm corresponding to the specific body part (202) based on the intraoperative imaging data (ID) and data indicating viewpoints corresponding to the plurality of intraoperative 2D images; c) estimating, by the computing device (10), based on the intraoperative imaging data (ID), the current position (5c) of the instrument (5) relative to the anatomical 3D shape (AS) of the specific body part (202); d) generating positioning guidance data (GD) by the computing device (10) comprising a visual representation of the estimated current position (5c) of the instrument (5) relative to the anatomical 3D shape (AS) of the specific body part (202); The computer-implemented method further comprising the steps of:

14. A computing device (10) comprising a processing unit (16) configured to perform the method of any one of claims 1 to 13.

15. A computing device (10) according to claim 14; an imaging device (20), such as at least one of a pre-operative imaging device, a post-operative imaging device, and an intra-operative imaging device, communicatively coupled to the computing device (10) and configured to capture 2D images of the particular body part of an anatomical object; A system (1) comprising:

16. A computer program product comprising instructions that, when executed by a processing unit (16) of a computing device (10), cause the computing device (10) to perform the method of any one of claims 1 to 13.

Images