Computer-implemented method, apparatus, system and computer program product for processing anatomical imaging data

Abstract

A computer-implemented method for processing anatomical imaging data, the method comprising, for a plurality of anatomical specimens: acquiring 3D imaging data of a particular body part of an anatomical specimen of the plurality of anatomical specimens; acquiring 2D images of the specific body part of the anatomical specimen from a plurality of different perspectives relative to the specific body part; determining a plurality of camera matrixes P corresponding to the obtained 2D image; projecting the acquired 3D imaging data according to the plurality of camera matrixes P to generate a synthesized 2D image; and associating the acquired 2D image with one of the synthesized 2D images as a training data pair according to the plurality of camera matrices P and a plurality of viewing angles. The method further includes generating a training data set for training an artificial intelligence algorithm using the training data pair.

Description

Technical Field

[0001] The present disclosure relates to a computer-implemented method, apparatus, system, and computer program product for processing anatomical imaging data. Specifically, the present disclosure relates to a computer-implemented method, apparatus, system, and computer program product for processing anatomical imaging data, including generating a training data set for training an artificial intelligence algorithm. Furthermore, the present disclosure relates to a computer-implemented method, apparatus, system, and computer program product for processing anatomical imaging data, including training an artificial intelligence algorithm using the training data set. Furthermore, the present disclosure relates to a computer-implemented method, apparatus, system, and computer program product for processing anatomical imaging data, including processing acquired 2D images of a particular body part of an anatomical object using the artificial intelligence algorithm.

[0002] The present invention also relates to a computer-implemented method of assisting in positioning a tool (such as a surgical tool) relative to a specific body part of a patient. The present invention also relates to a computing device configured to assist in positioning a tool (such as a surgical tool) relative to a specific body part of a patient. The present invention even relates to a system for assisting in positioning a tool relative to a specific body part of a patient. The present invention even relates to a computer program product comprising instructions that, when executed by a processing unit of a computing device, cause the computing device to assist in positioning a tool (such as a surgical tool) relative to a specific body part of a patient. Background Art

[0003] Despite the existence of modern computer-assisted surgery (CAS) solutions, the processing of anatomical imaging data, particularly for intraoperative navigation and guidance of complex orthopedic interventions, remains a challenging task to date. This is reflected in recent reports, which have estimated the clinical adoption rate of CAS solutions to be approximately 11% and 5% of surgical procedures performed. This becomes particularly sobering in light of the documented evidence of the benefits of CAS solutions in improving surgical accuracy and outcomes. The relatively low clinical adoption rate of CAS solutions has been attributed in the past to a number of factors, ranging from increased surgical time and operating costs to line-of-sight issues (specific to solutions requiring optical tracking systems) and, most importantly, the requirement for patient registration.

[0004] Patient registration is the process of aligning a preoperatively generated surgical plan with the patient's anatomy, and a variety of algorithms have been used in the prior art to address patient registration, such as feature-based registration, intensity-based registration, statistical shape modeling (SSM), and more recently, data-driven artificial intelligence registration methods. Despite the availability of advanced algorithms for patient registration, commercially available methods for processing anatomical imaging data (particularly CAS solutions) typically rely on basic approaches such as landmark-based methods. This is due to the fact that such basic methods can be fairly easily integrated into the CAS process, despite their low level of autonomy and versatility, while advanced registration methods, in contrast, suffer from various limitations, such as small capture range, lack of large training datasets, lack of robust similarity metrics for multimodal registration, and long computational times.

[0005] This has been the underlying reason for the emergence of algorithms that have evolved towards providing registration-free alternatives for processing anatomical imaging data. In the context of orthopedic surgery, such approaches typically rely solely on 2D imaging data that is ubiquitously available, for example, in the form of C-arm fluoroscopy. As an important component, these alternative solutions require processing of anatomical imaging data, such as directly reconstructing the anatomical structure of the anatomical object based solely on the 2D imaging data (e.g., C-arm imaging). Registration-free CAS can be achieved through full intraoperative cone-beam computed tomography (CBCT) or through more advanced algorithms that are able to reconstruct the 3D shape of the patient's anatomical structure using sparse fluoroscopy data. Due to reasons such as increased cost, time, and ionizing radiation, registration-free CAS solutions based on intraoperative CBCT imaging have not yet been widely adopted.

[0006] AI-based algorithms designed for processing anatomical imaging data, particularly anatomical 3D reconstruction based on X-ray input, do not suffer from the aforementioned limitations due to their ability to process anatomical imaging data, particularly to reconstruct patient anatomy based on a small number of ubiquitously available 2D images (particularly intraoperative X-rays). These AI-based data-driven approaches can be considered a stepping stone towards the development of registration-free methods for processing anatomical imaging data, such as future CAS systems.

[0007] However, a major technical bottleneck in creating such approaches is the limited availability of large-scale annotated anatomical imaging. Among various factors, ethical and radiation exposure concerns, along with the manual effort required for data annotation, are potential obstacles to collecting the required in vivo training datasets. To overcome these obstacles, most existing work has opted to create synthetic 2D datasets (e.g., X-rays), which can be automatically generated along with corresponding annotations. These synthetic datasets are typically created using digitally reconstructed radiography (DRR) techniques, which simulate planar X-ray projections based on input CT volumes with varying degrees of realism. Despite the flexibility and scalability of such datasets, they may not capture realistic anatomical imaging conditions, such as the actual radiological properties of anatomical structures, the noise characteristics of imaging equipment, and the backscattering behavior of real 2D imaging environments. If used to train downstream artificial intelligence models (such as 3D reconstruction networks), these synthetic datasets will inevitably result in models that perform well only on the synthetic training data and fail to achieve similar performance levels when applied to real (acquired) 2D images (e.g., X-ray images). This is often referred to as the domain gap phenomenon. Summary of the Invention

[0008] An object of the present disclosure is to provide a computer-implemented method, computing device, and computer program product for processing anatomical imaging data, including generating a training data set for training an artificial intelligence algorithm, which computer-implemented method, computing device, and computer program product do not have at least some of the disadvantages of the prior art.

[0009] In particular, an object of some embodiments disclosed herein is to provide a computer-implemented method, computing device, and computer program product for processing anatomical imaging data using a training dataset comprising paired real-synthetic X-ray images that can be acquired from anatomical specimens.

[0010] According to the present disclosure, these objects are solved by the features of the independent claims 1, 14, 15 and 16. Furthermore, further advantageous embodiments emerge from the dependent claims and the description.

[0011] This object is specifically addressed by a computer-implemented method for processing anatomical imaging data, the method comprising performing the following steps for a plurality of anatomical specimens: acquiring 3D imaging data of a specific body part of one of the plurality of anatomical specimens (referred to as acquired 3D imaging data); acquiring 2D images of the specific body part of the anatomical specimen from a plurality of different viewing angles relative to the specific body part (referred to as acquired 2D images); determining a plurality of camera matrices P corresponding to the acquired 2D images, wherein the camera matrices P indicate at least the viewing angles of the 2D images relative to the specific body part; generating a synthesized 2D image by projecting the acquired 3D imaging data according to the plurality of camera matrices P; and associating the acquired 2D image with one of the synthesized 2D images as a training data pair according to the plurality of camera matrices P. The method further comprises using the training data pair to generate a training dataset for training an artificial intelligence algorithm for processing the anatomical imaging data. Specifically, using the training data pair to generate the training dataset comprises storing the data pair with corresponding metadata and / or data indicating the association of the acquired 2D image with the synthesized 2D image.

[0012] According to some embodiments, one or more of the anatomical specimens are cadaver specimens, in particular human cadaver specimens.Alternatively or additionally, one or more of the anatomical specimens are living specimens, in particular human living specimens (humans).

[0013] According to some embodiments, the step of acquiring 3D imaging data of a specific body part of one of the multiple anatomical specimens includes: capturing the specific body part of the one anatomical specimen by a 3D imaging device (such as a computed tomography (CT) imaging device). Alternatively or additionally, the step of acquiring 3D imaging data of a specific body part of one of the multiple anatomical specimens includes: receiving 3D imaging data (capturing the specific body part of the one anatomical specimen) from a communicatively connected 3D imaging device by a computing device. Alternatively or additionally, the step of acquiring 3D imaging data of a specific body part of one of the multiple anatomical specimens includes: retrieving 3D imaging data from a communicatively connected database by a computing device, the database storing 3D imaging data of multiple anatomical specimens.

[0014] According to some embodiments, the step of acquiring 2D images of the specific body part of the anatomical specimen from a plurality of different viewing angles relative to the specific body part comprises: capturing, by a 2D imaging device, the 2D images of the specific body part of the anatomical specimen from a plurality of different viewing angles relative to the specific body part. Alternatively or additionally, the step of acquiring 2D images of the specific body part of the anatomical specimen from a plurality of different viewing angles relative to the specific body part comprises: receiving, by a computing device, the 2D images from a communicatively connected 2D imaging device. Alternatively or additionally, the step of acquiring 2D images of the specific body part of the anatomical specimen from a plurality of different viewing angles relative to the specific body part comprises: retrieving, by a computing device, the 2D images from a communicatively connected database that stores the 2D images of the specific body part of the anatomical specimen from a plurality of different viewing angles relative to the specific body part.

[0015] Determining a plurality of camera matrices P corresponding to the acquired 2D images includes determining intrinsic and extrinsic camera properties of the acquired 2D images. Intrinsic camera properties relate to properties of a 2D imaging device that captured the 2D images, such as focal length and principal point position on an image plane. Extrinsic camera properties relate to the position and orientation of a camera relative to the specific body part, in other words, its viewing angle.

[0016] After determining the camera matrix P corresponding to the acquired 2D images (at least indicating the viewing angle of the 2D images relative to the specific body part), a synthesized 2D image is generated by the computing device by projecting the acquired 3D imaging data according to the plurality of camera matrices P. In other words, the synthesized 2D images are generated as if they were captured by the same 2D imaging device as the acquired 2D images and from the same viewing angle (the same camera matrix). The technical consideration behind this method is that, due to inevitable inaccuracies in intrinsic and extrinsic camera properties, it is virtually impossible to precisely match the camera matrix P with the prescribed camera matrix (of the synthesized 2D images) when acquiring 2D images of anatomical specimens. Instead of attempting to correct for inaccuracies in the intrinsic and extrinsic camera properties when capturing the 2D images of the anatomical specimens, according to the present invention, the generation of the synthesized 2D images is performed based on the camera matrix of the acquired 2D images, which is free of such inaccuracies.

[0017] In other words, when generating training data pairs, instead of trying to position the device of the 2D image to match the camera matrix (particularly the perspective of the synthesized 2D image), the 2D image is synthesized based on the camera matrix P of the acquired 2D image.

[0018] The training data sets generated using the training data pairs are particularly suitable for training artificial intelligence algorithms for processing anatomical imaging data. Generating the training data sets takes into account specific technical considerations for providing training data pairs, wherein acquired 2D images are already associated with synthesized 2D images according to a corresponding camera matrix P. Specifically, one or more of the acquired 2D images are associated with a synthesized 2D image generated using the corresponding camera matrix P (particularly using a corresponding viewing angle). In this way, the training data pairs (including the associated acquired 2D images and synthesized 2D images) are not only suitable but also particularly suitable for training artificial intelligence algorithms, as each data pair includes an acquired 2D image to be provided as input and a synthesized 2D image as a reference for an expected / desired output of the artificial intelligence algorithm, or vice versa, i.e., a synthesized 2D image as input and an acquired 2D image as a reference for an expected / desired output. By providing corresponding data pairs, artificial intelligence algorithms can be trained by applying a suitable objective function, cost function, or loss function (comparing the generated output with the expected output).

[0019] In order to allow accurate determination of a plurality of camera matrices P corresponding to the acquired 2D images and to allow projection of the acquired 3D imaging data according to the plurality of camera matrices P, according to some embodiments, the anatomical specimen is provided (assembled) with a calibration device before acquiring the 3D imaging data and the 2D images of the specific body part of the anatomical specimen. In the case of a live anatomical specimen, a registration device is used as a calibration device that is non-invasively or minimally invasively attached to the anatomical specimen. In the case of an ex vivo cadaveric anatomical specimen, a reference device is attached to the anatomical specimen as a calibration device.

[0020] After the anatomical specimen has been provided with a calibration device, 2D coordinates of the calibration device are determined in each of the 2D images (referred to as determined 2D coordinates).

[0021] Furthermore, after the anatomical specimen has been placed with the calibration device, the 3D coordinates of the calibration device are determined based on the 3D imaging data. Thereafter, a 2D-3D coordinate correspondence is determined between the 2D coordinates of the calibration device in each of the 2D images and the 3D coordinates of the calibration device. The 2D-3D coordinate correspondences are further used to determine the plurality of camera matrices P corresponding to the acquired 2D images. In other words, by determining the coordinates of the calibration device in both the acquired 2D and 3D images, the camera matrix P can be accurately determined, and the 3D image data can be projected to produce a composite 2D image that accurately matches the camera matrix P of the acquired 2D image.

[0022] According to some embodiments, in order to avoid that the calibration device is captured in the 3D imaging data and thus projected onto the synthesized 2D image, further 3D imaging data is acquired before the anatomical specimen is provided with the calibration device. In order to compensate for eventual alignment errors, a 3D-3D registration of the 3D imaging data acquired before the anatomical specimen is provided with the calibration device with the 3D imaging data acquired after the anatomical specimen is provided with the calibration device is performed by the computing device. After the 3D-3D registration of the 3D imaging data acquired before the anatomical specimen is provided with the calibration device with the 3D imaging data acquired after the anatomical specimen is provided with the calibration device, the synthesized 2D image is generated by projecting the 3D imaging data acquired before the anatomical specimen is provided with the calibration device according to the multiple camera matrices P. In this way, the calibration device is not present in the synthesized 2D image (but only the anatomical specimen is present).

[0023] Although the calibration device is useful for determining the camera matrix P of the acquired 2D image, it is not desirable for the calibration device to be present in the acquired 2D image. Therefore, in order to remove the calibration device captured in the 2D imaging data, according to some embodiments, the projection of the calibration device in the 2D image is inpainted (removed) from the acquired 2D image before the acquired 2D image is used to generate the training dataset.

[0024] According to some embodiments, determining a plurality of camera matrices P corresponding to the acquired 2D images includes: detecting 2D coordinates of the calibration device on the acquired 2D images; detecting 3D coordinates of the calibration device on the acquired 3D imaging data; determining a 2D-3D coordinate correspondence between the 2D coordinates and the 3D coordinates of the calibration device; and determining the camera matrix P using the 2D coordinates and the 3D coordinates of the calibration device and the 2D-3D coordinate correspondence. Specifically, determining the 2D-3D coordinate correspondence between the 2D coordinates and the 3D coordinates of the calibration device is performed in an iterative process, wherein, for each possible correspondence between the 2D coordinates and the 3D coordinates of the calibration device: estimating the camera matrix P; projecting the 3D imaging data according to the estimated camera matrix P; and calculating a reprojection error by comparing the projected 3D imaging data with the corresponding acquired 2D images. After the reprojection errors have been calculated for all possible correspondences between the 2D coordinates and the 3D coordinates of the calibration device, the camera matrix P of the acquired 2D image is determined using the 2D-3D coordinate correspondence with the lowest reprojection error.

[0025] According to some embodiments, the computer-implemented method of the present invention further comprises: obtaining a pre-training dataset for processing anatomical imaging data comprising a plurality of synthetic 2D images capturing the particular body part of the anatomical specimen; and pre-training an artificial intelligence algorithm (referred to as a pre-trained artificial intelligence algorithm) using the pre-training dataset for processing anatomical imaging data. Given the limited availability of annotated anatomical imaging, according to a specific embodiment, the pre-training dataset comprises synthetic 2D images, such as fluoroscopic shots (i.e., DRRs) generated using different camera matrices, particularly given that the input 3D image data (such as CT scans) is generated from different perspectives. For example, using this method, up to hundreds of annotated "synthetic" 2D images (which capture the particular body part) can be generated from a single annotated CT scan, and these annotated "synthetic" 2D images can be used to train an artificial intelligence algorithm for processing anatomical images, such as to reconstruct accurate anatomical 3D shapes from as few 2D intraoperative images as possible.

[0026] In view of the limitations and shortcomings of the domain gap described in the "Background Art" section of this application, another object of some embodiments disclosed herein is to provide a domain adaptation framework method, using which an artificial intelligence algorithm pre-trained only on synthetic images can be used to process acquired (real) 2D images (e.g., X-rays). For example, such a method can be used to reconstruct the anatomical 3D shape (AS) of the specific body part of an anatomical object based on the 2D image of the specific body part and data indicating a camera matrix (in particular, indicating the viewing angle corresponding to the 2D image); and / or for anatomical segmentation of the 2D image of the specific body part of the anatomical object.

[0027] This objective is achieved by applying a transfer learning process, i.e., training a pre-trained artificial intelligence algorithm (also called retraining) using a training dataset comprising data pairs of synthesized 2D images and acquired (real) 2D images of the corresponding camera matrix P.

[0028] According to some embodiments, training the pre-trained artificial intelligence algorithm (also referred to as retraining) includes fixing a first subset of network parameters of network components of the pre-trained artificial intelligence algorithm (such as weights and biases of an artificial neural network), and adjusting a second subset of network parameters using the training data pairs of the training dataset as input. For one specific embodiment of reconstructing the anatomical 3D shape of a specific body part of an anatomical object, retraining the pre-trained artificial intelligence algorithm includes fixing network parameters of network components other than a 2D feature extractor module of the pre-trained artificial intelligence algorithm (such as weights and biases of an artificial neural network), and adjusting the network parameters of the 2D feature extractor module using the training data pairs of the training dataset as input.

[0029] As a practical application, according to the present invention, the computer-implemented method further comprises: acquiring a 2D image of the specific body part of the anatomical object, and processing the acquired 2D image using an artificial intelligence algorithm trained using the training data set.

[0030] In this context, the term "anatomical object" refers to a specific object of interest - such as a patient during surgery or a medical examination - as opposed to a plurality of "anatomical specimens" that are analyzed to obtain the training dataset.

[0031] Alternatively, the purpose of providing a domain adaptation framework is solved by a style transfer process, using which an artificial intelligence algorithm that has been pre-trained on a synthetic image can be employed to process an acquired (real) 2D image (e.g., an X-ray). Specifically, according to some embodiments, the purpose of providing a domain adaptation framework is solved because the computer-implemented method further comprises: training a style transfer algorithm using the training dataset; and applying the style transfer process to the acquired 2D image using the style transfer algorithm. The style transfer process is applied to the acquired 2D image (referred to as a pre-processed acquired 2D image) before processing the acquired 2D image of the anatomical object using the pre-trained artificial intelligence algorithm.

[0032] In other words, the domain gap (pre-training performed using a large amount of available but synthetic 2D images) is filled because the input images (acquired 2D images of anatomical objects) are pre-processed by a style transfer algorithm before being fed as input to the pre-trained artificial intelligence algorithm.

[0033] Specifically, the style transfer process includes extracting texture information from the synthesized 2D image of the training dataset; and transferring the texture information to the acquired 2D image of the anatomical object while preserving the underlying semantic content of the acquired 2D image. In other words, the acquired 2D image (of the anatomical object) is made to look like the synthesized 2D image that the pre-trained artificial intelligence algorithm has been trained to process.

[0034] A unique feature of the style transfer process of the present invention is the development of a model for converting an acquired 2D image of an anatomical object (e.g., real X-ray data) into a synthetic 2D image (e.g., a synthetic DRR image). Compared to known methods of converting a synthetic 2D image (e.g., a synthetic DRR image) into an image similar to an acquired (e.g., similar to an actual X-ray), where the target domain is highly heterogeneous, the reverse transfer direction of the present invention is implemented to ensure a homogeneous target domain (e.g., by controlling the synthetic DRR space). In addition, for a variety of applications (e.g., segmentation, reconstruction, landmark detection, etc.) for which artificial intelligence algorithms pre-trained on synthetic data are already available, one can use the style transfer process of the present invention in a sequential manner, where the acquired 2D image (e.g., real X-ray) input is first converted to the synthetic domain, which is then fed into the pre-trained artificial intelligence algorithm.

[0035] In a specific embodiment, the style transfer process implements one or more of the following methods:

[0036] -Encoder-Decoder Deep Convolutional Network EDDC:

[0037] -Neural Style Transfer NST algorithm:

[0038] -Deep Convolutional Generative Adversarial Network DCGAN:

[0039] -Cycle-consistent generative adversarial network CycleGAN.

[0040] As a practical application, according to the present invention, the computer-implemented method further includes: acquiring a 2D image of the specific body part of an anatomical object; applying the style transfer process to the acquired 2D image using the style transfer algorithm; and processing the preprocessed acquired 2D image using a pretrained artificial intelligence algorithm.

[0041] As a side note, in the context of this disclosure, the terms "pre-training," "pre-trained," refer to the training of an artificial intelligence algorithm using training data other than the training dataset generated as part of the present invention.

[0042] As a first use case, according to some embodiments of the present invention, processing the acquired 2D image of the specific body part of the anatomical object using the artificial intelligence algorithm includes: reconstructing the anatomical 3D shape of the specific body part of the anatomical object based on the acquired 2D image of the specific body part and data indicating the viewing angle corresponding to the acquired 2D image.

[0043] As another use case, according to some embodiments of the present invention, processing the acquired 2D image of the specific body part of the anatomical object using the artificial intelligence algorithm includes: performing anatomical segmentation on the acquired 2D image to identify one or more anatomical characteristics of the specific body part.

[0044] The present application also relates to a computing device comprising a processing unit configured to perform a method according to one of some embodiments disclosed herein. According to some embodiments, the computing device comprises any one or a combination of the following: a central processing unit (CPU), a physical or virtual computer, a local and / or remote portable, desktop and / or server computer, a cloud-based server or software as a service, and / or a dedicated hardware circuit system, such as an ASIC or FPGA.

[0045] The present application also relates to a system comprising: a computing device according to one of some embodiments disclosed herein; and an imaging device, such as a pre-operative, post-operative and / or intra-operative imaging device, which is communicatively connected to the computing device and configured to capture a 2D image of the specific body part of the anatomical object.

[0046] The present application also relates to a computer program product comprising instructions which, when executed by a processing unit of a computing device, cause the computing device to perform a method according to any one of the embodiments disclosed herein.

[0047] According to some embodiments of the present invention, the training dataset, the pre-trained artificial intelligence algorithm, the trained artificial intelligence algorithm, and / or the style transfer algorithm are each generated or trained for a specific body part. Correspondingly, their use for training or processing acquired 2D images is intended for the same specific body part.

[0048] Another object of the present invention is to provide a method, computing device, system and computer program product for assisting in positioning a tool relative to a specific body part of a patient, which overcomes one or more of the shortcomings of the prior art.

[0049] Specifically, an object of the present invention is to provide a registration-free method for assisting in positioning a tool relative to a specific body part of a patient, which method can use only intraoperative 2D imaging data (i.e., no navigation hardware needs to be installed in the operating room) to reconstruct the anatomical 3D shape and generate a visual representation of the positioning of the tool relative to the specific body part of the patient.

[0050] According to the present disclosure, this object is achieved by the features of independent claim 1. Furthermore, further advantageous embodiments emerge from the dependent claims and the description.

[0051] In particular, this object is achieved by a computer-implemented method for assisting in positioning a tool, such as a surgical tool (e.g., a surgical drill, a knife, or surgical laser equipment) or a medical diagnostic tool, relative to a specific body part of a patient, the method comprising:

[0052] Receive intraoperative 2D imaging data;

[0053] reconstructing an anatomical 3D shape using the intraoperative 2D imaging data and using an artificial intelligence algorithm corresponding to the specific body part;

[0054] estimating a current position of the tool based on the intraoperative 2D imaging data; and

[0055] Positioning guidance data is generated, the positioning guidance data comprising a visual representation of the estimated current position of the tool relative to the anatomical 3D shape of the particular body part.

[0056] In a specific embodiment, in preparation for / before surgical treatment of the patient, the following steps are repeatedly or continuously performed over a period of time: receiving intraoperative 2D imaging data, estimating the current positioning of the tool; and generating guidance data.

[0057] Receive intraoperative 2D imaging data

[0058] Intraoperative 2D imaging data is received by a computing device from an intraoperative imaging device arranged in the vicinity of a patient. Here, the intraoperative imaging device being arranged in the vicinity of the patient refers to a positioning that allows the intraoperative imaging device to capture intraoperative 2D imaging data of the patient. The imaging data includes a plurality of intraoperative 2D images. Two or more of the intraoperative 2D images capture the specific body part of the patient from two or more different perspectives relative to the specific body part of the patient. One or more of the same plurality of intraoperative 2D images capturing the specific body part also captures at least a portion of the tool from at least one perspective. In the context of the present invention, the term "perspective" (perspective with respect to the imaging data) refers to the position (e.g., in an x, y, and z Cartesian coordinate system) and / or orientation (e.g., roll, pitch, heading) of the intraoperative imaging device relative to the specific body part or relative to the tool.

[0059] According to some embodiments of the present invention, the intraoperative 2D imaging data includes one or more of the following: a) radiation-based images, in particular X-ray images; b) ultrasound images; c) arthroscopic images; d) optical images; and / or e) any other cross-sectional images.

[0060] An image of the specific body part of the patient or an image of a portion of the tool is captured using an intraoperative imaging device that is communicatively connected to the computing device. In the case of radiation-based images, the intraoperative imaging device may include a C-arm intraoperative imaging device based on X-ray technology. The C-arm intraoperative imaging device includes a generator (X-ray source) and an image intensifier or a flat panel detector. The C-shaped connecting element allows movement horizontally, vertically and / or around a rotation axis so that a 2D X-ray image of the patient can be generated from various viewing angles around the patient. The generator emits X-rays that penetrate the patient's body. The image intensifier or detector converts the X-rays into a visible image that is transmitted to the computing device.

[0061] The intraoperative 2D imaging data includes data indicating a viewing angle corresponding to the plurality of intraoperative 2D images, the viewing angle identifying a position and / or orientation of an intraoperative imaging device that captured the plurality of intraoperative 2D images (such as a position of the intraoperative imaging device in an x, y, and z Cartesian coordinate system relative to the particular body part and / or an orientation such as roll, pitch, and heading). According to some embodiments disclosed herein, the data indicating the viewing angle corresponding to the plurality of intraoperative 2D images is stored in a data storage device included by the computing device or communicatively connected to the computing device. Alternatively or additionally, the viewing angle corresponding to the intraoperative 2D imaging data is estimated by the computing device based on the intraoperative 2D imaging data.

[0062] In one embodiment, estimating the viewing angle corresponding to the intraoperative 2D imaging data is performed using a tool geometric model indicating the geometry of the tool. First, a plurality of projections of the tool geometric model are calculated based on a plurality of candidate viewing angles. The candidate viewing angles are selected as discrete viewing angles within a predefined space of possible viewing angles of the intraoperative imaging device. In other words, unrealistic positions and orientations of the intraoperative imaging device relative to the patient are not taken into account to save computing power. Thereafter, the viewing angle corresponding to the intraoperative 2D image of the intraoperative 2D imaging data is identified by comparing at least a portion of the tool as captured by the corresponding 2D image of the intraoperative 2D imaging data with the plurality of projections calculated based on the plurality of candidate viewing angles. Specifically, the comparison includes applying a matching function to identify the best match between one of the projections of the "virtual" tool (based on the tool geometric model) calculated based on the plurality of candidate viewing angles and the portion of the "physical" tool as captured by the intraoperative imaging device. The candidate viewing angle that produces the best match is selected as the estimated viewing angle.

[0063] According to some embodiments disclosed herein, estimating the view angle corresponding to the intraoperative 2D imaging data is performed using an artificial intelligence algorithm that is trained using a large number of imaging datasets with known view angles. To overcome limitations in the availability and / or accuracy of imaging datasets with known view angles, a large number of imaging datasets comprising intraoperative 2D images from known view angles are generated based on 3D imaging data (particularly computed tomography (CT) scans). Using this artificial intelligence algorithm trained prior to the surgical procedure, the intraoperative positioning of the intraoperative imaging device can be estimated based solely on the intraoperative images, without the need for external tracking devices or calibration phantoms.

[0064] Generate anatomical 3D shapes

[0065] The anatomical 3D shape of the specific body part is reconstructed by the computing device using an artificial intelligence algorithm corresponding to the specific body part based on the intraoperative 2D imaging data and data indicating viewing angles corresponding to the multiple intraoperative 2D images.

[0066] According to some embodiments disclosed herein, the anatomical 3D shape is reconstructed as a voxelized volume and / or mesh. It is important to emphasize that the artificial intelligence algorithm must be a model corresponding to the specific body part, thereby enabling the reconstruction of the 3D anatomical shape from multiple intraoperative 2D images of the specific body part.

[0067] According to specific embodiments disclosed herein, the artificial intelligence algorithm is trained using a large number of annotated imaging datasets that capture a body part corresponding to the specific body part of the patient (possibly of a person other than the patient). The annotations of the imaging datasets include data that identifies and / or describes properties of the body part, such as data that identifies pixels, vectors, contours, surfaces within an intraoperative 2D image, and / or voxels within a 3D image that captures the specific body part.

[0068] In order to overcome the limitations in the availability and / or accuracy of annotated imaging datasets that capture body parts corresponding to the specific body part of the patient - according to some embodiments disclosed herein - a large number of annotated imaging datasets are generated based on annotated 3D imaging data (particularly computed tomography (CT) scans), wherein the annotated 3D imaging data captures body parts corresponding to the specific body part of the patient. Specifically, synthetic intraoperative 2D images (such as fluoroscopic images (i.e., DRRs)) are generated from different perspectives around the patient taking into account the input preoperative CT scan. For example, using this approach, up to hundreds of annotated "synthetic" intraoperative 2D images (which capture the specific body part) can be generated based on a single annotated CT scan, and the annotated "synthetic" intraoperative 2D images can be used by the artificial intelligence algorithm to improve its ability to reconstruct accurate anatomical 3D shapes based on as few 2D intraoperative images as possible.

[0069] Estimate the current location of the tool

[0070] After the anatomical 3D shape has been reconstructed, the current position of the tool relative to the anatomical 3D shape of the specific body part is estimated based on the intraoperative 2D imaging data (in particular, the intraoperative 2D image of the imaging data capturing the tool). According to some embodiments disclosed herein, estimating the current position of the tool is performed using a tool geometric model indicating the geometric shape of the tool. First, a projection of the tool geometric model is compared with at least a portion of the tool as captured by the corresponding 2D image of the intraoperative 2D imaging data. The tool geometric model is projected onto the plane of one or more 2D images of the intraoperative 2D imaging data capturing at least a portion of the tool. The plane of the intraoperative 2D image of the intraoperative 2D imaging data is determined based on the viewing angle of each 2D image. Thereafter, a positioning of the tool geometric model resulting from the projection onto the plane of the intraoperative 2D image of the intraoperative 2D imaging data is determined, the positioning being (best) matched to at least a portion of the tool as captured by the corresponding 2D image of the intraoperative 2D imaging data. In other words, an inverse process is applied compared to the (initial) determination of the viewing angle of the intraoperative 2D imaging data. However, this inverse process is not necessarily applied to the same intraoperative 2D image (of the intraoperative 2D imaging data) as the intraoperative 2D image used to determine the viewing angle of the image used to reconstruct the anatomical 3D shape.

[0071] According to some embodiments disclosed herein, although the anatomical 3D shape is reconstructed once in an initial stage of the method for assisting in positioning a tool, the estimation of the current positioning of the tool is repeatedly performed at set intervals and / or triggered by certain events and / or manually triggered.

[0072] In order to improve the accuracy of estimating the positioning of the tool and / or improve the accuracy of estimating the viewing angle corresponding to the intraoperative 2D imaging data, according to other embodiments, the method of the present invention further comprises: providing the tool according to a tool geometric model. The tool geometric model is specifically designed to optimize the estimation of its positioning based on as few intraoperative images as possible. Specifically, the tool is designed so that at least a portion thereof is not completely rotationally symmetric around any axis of a Cartesian coordinate system, so as to allow the orientation of the tool to be estimated based on the intraoperative 2D images. Alternatively or additionally, the tool is designed to include special markings to facilitate its identification based on the 2D intraoperative images.

[0073] Generate positioning guidance data

[0074] After the anatomical 3D shape of the particular body part has been reconstructed and the current positioning of the tool has been estimated, positioning guidance data is generated by the computing device, the positioning guidance data comprising a visual representation of the estimated current positioning of the tool relative to the anatomical 3D shape of the particular body part. According to some embodiments disclosed herein, the guidance data is generated as a 2D image to be displayed on a computer display. Alternatively or additionally, the guidance data is generated as an augmented reality overlay, comprising overlay metadata that allows an augmented reality device, such as a headset, to project the overlay onto a user's field of view such that the overlay is aligned with the user's view of the patient's particular body part and / or with the user's view of the tool. According to some embodiments disclosed herein, the visual representation of the estimated current positioning of the tool is superimposed on the visual representation of the reconstructed anatomical 3D shape.

[0075] According to some embodiments disclosed herein, the computing device controls a display device to display at least a portion of the guidance data, wherein the display device is a computer screen, an augmented reality head-mounted device, or any device configured to display guidance data.

[0076] In order to guide the surgeon in correctly positioning the tool, according to further embodiments disclosed herein, a prescribed position of the tool relative to the anatomical 3D shape of the particular body part is identified by the computing device, and a visual representation of the prescribed position of the tool is superimposed on a visual representation of the estimated current position of the tool. The prescribed position of the tool is retrieved or received by the computing device from a data storage device, the computing device including the data storage device or the data storage device being communicatively connected to the computing device. Alternatively or additionally, the prescribed position of the tool is calculated by the computing device, the prescribed position of the tool being determined by an optimization function based on the anatomical 3D shape of the body part and data indicative of the surgical procedure.

[0077] Some embodiments disclosed herein are advantageous because they enable automated pre-operative surgical planning based on the reconstructed anatomical 3D shape and guide the surgeon in the placement of surgical tools. Given that the anatomical 3D shape of the body part (e.g., spine) is reconstructed using intraoperative 2D imaging data, a prescribed positioning / trajectory of the tool can be identified based on the anatomical 3D shape, eliminating the need for a pre-operative planning phase to define a safe implant trajectory and the need to register the pre-operative data to the intra-operative patient's positioning.

[0078] Another object of the present invention is to provide a computing device for positioning a tool relative to a specific body part of a patient, which can reconstruct the anatomical 3D shape and generate a visual representation of the positioning of the tool relative to the specific body part of the patient using only intraoperative 2D imaging data (i.e., no navigation hardware needs to be installed in the operating room and no preoperative planned alignment process needs to be performed).

[0079] The above-mentioned object is also achieved by a computing device, comprising: a data input interface; a data output interface; a processing unit; and a storage unit. The data input interface, such as a wired (e.g., Ethernet, DVI, HDMI, VGA) data communication interface and / or a wireless data communication interface (e.g., 4G, 5G, Wi-Fi, Bluetooth, ultra-wideband), is communicatively connected to an intraoperative imaging device and is configured to receive intraoperative 2D imaging data therefrom. The data output interface, such as a wired (e.g., Ethernet, DVI, HDMI, VGA) data communication interface and / or a wireless data communication interface (e.g., 4G, 5G, Wi-Fi, Bluetooth, ultra-wideband, infrared), is configured to transmit at least a portion of the guidance data to a display device communicatively connected to the data output interface. The storage unit comprises instructions that, when executed by the processing unit, cause the computing device to perform a method for assisting in positioning a tool according to any one of the embodiments disclosed herein.

[0080] According to some embodiments, the computing device is a standalone computer communicatively connected to the intraoperative imaging device. Alternatively or additionally, the computing device is a remote computer (e.g., a cloud-based computer) communicatively connected to the intraoperative imaging device using a communication network (particularly at least partially using a mobile communication network). Alternatively or additionally, the computing device is integrated into the intraoperative imaging device or the display device.

[0081] Another object of the present invention is to provide a system for positioning a tool relative to a specific body part of a patient, which can use only intraoperative 2D imaging data (i.e., no navigation hardware needs to be installed in the operating room and no preoperatively planned alignment process needs to be performed) to reconstruct the anatomical 3D shape and generate a visual representation of the positioning of the tool relative to the specific body part of the patient.

[0082] The above-mentioned object is also achieved by a system, comprising: a computing device according to any of the embodiments disclosed herein; an intraoperative imaging device; and a display device, the system being configured to perform a method according to any of the embodiments disclosed herein. The intraoperative imaging device is communicatively connected to the computing device and is arranged in proximity to a patient, thereby allowing the intraoperative imaging device to capture intraoperative 2D imaging data of the patient, such that two or more of the intraoperative 2D images capture the patient's specific body part from two or more different perspectives relative to the patient's specific body part. One or more of the same plurality of intraoperative 2D images capturing the specific body part also capture at least a portion of the tool from at least one perspective. In the case of radiation-based images as intraoperative images, the intraoperative imaging device comprises a C-arm intraoperative imaging device based on X-ray technology. The C-arm intraoperative imaging device comprises a generator (X-ray source) and an image intensifier or a flat-panel detector. A C-shaped connecting element allows for movement horizontally, vertically, and / or about a rotational axis, so that 2D X-ray images of the patient can be generated from various perspectives around the patient. The generator emits X-rays that penetrate the patient's body. The image intensifier or detector converts the X-rays into a visible image that is transmitted to the computing device.The display device is a computer screen, an augmented reality headset, or any device configured to display guidance data.

[0083] Another object of the present invention is to provide a computer program product for positioning a tool relative to a specific body part of a patient, which can reconstruct the anatomical 3D shape and generate a visual representation of the positioning of the tool relative to the specific body part of the patient using only intraoperative 2D imaging data (i.e., no navigation hardware needs to be installed in the operating room and no preoperatively planned alignment process needs to be performed).

[0084] The above objects are solved by a computer program product, which comprises instructions, which, when executed by a processing unit of a computing device, cause the computing device to perform a method according to any one of the embodiments disclosed herein.

[0085] According to some embodiments, the instructions (comprising the computer program product) comprise an artificial intelligence algorithm corresponding to a particular body part of a patient, the artificial intelligence algorithm having been trained using a large number of annotated imaging datasets capturing a body part corresponding to the particular body part of the patient, wherein the annotations comprise data identifying and / or describing attributes of the body part.

[0086] According to some embodiments, the instructions (comprising the computer program product) include instructions for controlling an intraoperative imaging device to capture intraoperative 2D imaging data, the intraoperative 2D imaging data comprising intraoperative 2D images, a plurality of intraoperative 2D images capturing the specific body part of the patient from a plurality of different perspectives relative to the specific body part of the patient, and one or more of the plurality of intraoperative 2D images capturing at least a portion of the tool from at least one perspective.

[0087] According to some embodiments, the instructions (comprising the computer program product) include instructions for controlling a display device, such as to display at least a portion of the guidance data, the guidance data comprising a visual representation of an estimated current position of the tool, a visual representation of a reconstructed anatomical 3D shape, and / or a visual representation of a prescribed position of the tool.

[0088] It should be understood that both the foregoing general description and the following detailed description present several embodiments and are intended to provide an overview or framework for understanding the nature and character of the present disclosure. The accompanying drawings are included to provide a further understanding and are incorporated into and constitute a part of this specification. The drawings illustrate various embodiments and, together with the description, serve to explain the principles and operation of the disclosed concepts.

[0089] As used in this specification, the term "particular" refers to some embodiments of the invention without indicating any preference or indicating that a feature introduced as "particular" will be essential to all embodiments of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The present disclosure will be explained in more detail by way of example with reference to the accompanying drawings, in which:

[0091] Figure 1 A flow chart illustrating the steps of a method according to the present invention for processing anatomical imaging data, including generating a training data set for training an artificial intelligence algorithm;

[0092] Figure 2 A flowchart illustrating the steps of a process for determining a camera matrix;

[0093] Figure 3 a data collection and processing flow for a specific application of the method for processing anatomical imaging data according to the present invention on fresh-frozen ex vivo whole-torso specimens;

[0094] Figure 4 ( Figure 3 an exemplary 2D image during a process of restoring a calibration device from a 2D image;

[0095] Figure 5 A flow chart illustrating the steps of a method for processing anatomical imaging data by applying a transfer learning process, including generating a training dataset, pre-training an artificial intelligence algorithm using the pre-trained dataset, training the pre-trained artificial intelligence algorithm using the training dataset, and processing the anatomical imaging data using the trained artificial intelligence algorithm;

[0096] Figure 6 An illustration of a specific embodiment of retraining a pre-trained artificial intelligence algorithm by applying a transfer learning process;

[0097] Figure 7 A flow chart illustrating the steps of a method for processing anatomical imaging data by applying a style transfer process, including generating a training dataset, pre-training an artificial intelligence algorithm using the pre-training dataset, training a style transfer algorithm using the training dataset, pre-processing the anatomical imaging data using the style transfer algorithm, and processing the pre-processed anatomical imaging data using the pre-trained artificial intelligence algorithm;

[0098] Figure 8 A flowchart illustrating the steps of applying a style transfer algorithm to anatomical imaging data;

[0099] Figure 9 An illustration of a first implementation of a style transfer algorithm based on a deep convolutional generative adversarial network (DCGAN);

[0100] Figure 10 An illustration of a second implementation of a style transfer algorithm based on a cycle-consistent generative adversarial network (CycleGAN);

[0101] Figure 11 A highly schematic perspective view of a system for assisting in positioning a tool, such as that installed in an operating room, according to one embodiment of the present invention;

[0102] Figure 12 A flow chart illustrating the steps of a method of assisting in positioning a tool according to one embodiment of the present invention;

[0103] Figure 13 A flow chart illustrating steps for reconstructing an anatomical 3D shape based on intraoperative 2D imaging data and data indicating a camera matrix corresponding to a plurality of intraoperative 2D images according to one embodiment of the present invention;

[0104] Figure 14 Schematic illustration of segmentation of intraoperative 2D imaging data to identify specific body parts of a patient;

[0105] Figure 15A schematic illustration of another embodiment of segmenting intraoperative 2D imaging data includes identifying a region of interest followed by semantic segmentation of the region of interest to identify a specific body part of a patient within the region of interest;

[0106] Figure 16 Schematic illustration of reconstruction of anatomical 3D shapes based on segmented intraoperative 2D imaging data using artificial intelligence algorithms corresponding to specific body parts;

[0107] Figure 17 A flow chart illustrating the steps of another embodiment of reconstructing an anatomical 3D shape in multiple stages;

[0108] Figure 18 Schematic illustration of the prescribed positioning of the determination tool;

[0109] Figure 19A an illustrative embodiment of a visual representation of an estimated current position of a tool and a visual representation of a prescribed position of the tool superimposed on a visual representation of a reconstructed anatomical 3D shape;

[0110] Figure 19B An illustrative embodiment of a visual representation of an estimated current position of a tool superimposed onto a 2D image of the intraoperative 2D imaging data ID; and

[0111] Figure 19C An illustrative embodiment of a visual representation of an estimated current position of a tool onto a visual representation of a reconstructed anatomical 3D shape, a visual representation of a prescribed position of the tool, and a visual representation of an ideal screw trajectory for a surgical implant. DETAILED DESCRIPTION

[0112] Reference will now be made in detail to certain embodiments, examples of which are illustrated in the accompanying drawings, in which some, but not all, features are shown. Indeed, some of the embodiments disclosed herein may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Whenever possible, the same reference numerals are used to refer to the same components or parts.

[0113] Figure 1 The steps of the method for processing anatomical imaging data according to the present invention are illustrated in FIG, comprising generating a training data set for training an artificial intelligence algorithm.

[0114] In a first step S10, 3D imaging data of a particular body part of one of a plurality of anatomical specimens is acquired (referred to as acquired 3D imaging data). In a first sub-step S12 of step S10, 3D imaging data of the plurality of anatomical specimens is acquired using a 3D imaging device, particularly a computed tomography (CT) scan. In an optional sub-step S14 of step S10, individual body parts, such as vertebrae, are segmented in the 3D volume.

[0115] In step S30, after or simultaneously with step S10, 2D images are acquired. In sub-step S32, 2D images (referred to as acquired 2D images) of the specific body part of the anatomical specimen are acquired from a plurality of different viewing angles relative to the specific body part.

[0116] In step S40, a plurality of camera matrices P corresponding to the acquired 2D images are determined, wherein the camera matrix P at least indicates the viewing angle of the 2D images relative to the specific body part. Figure 2 The details of determining the multiple camera matrices P are described. Finally, in step S50, a training data set is generated. In a first sub-step S52 of step S50, a plurality of camera matrices P corresponding to the acquired 2D images are determined by the computing device 10. As another sub-step S54 of step S50 (generating a training data set), a synthetic 2D image is generated by the computing device by projecting the acquired 3D imaging data according to the multiple camera matrices P. Thus, a synthetic 2D image is generated for each camera matrix P (i.e., for each acquired 2D image viewing angle). As another sub-step S56 of step S50, the acquired 2D image is associated with one of the synthetic 2D images according to the multiple camera matrices P (in particular, multiple viewing angles) as a training data pair. In other words, each of the 2D images is associated with a synthetic 2D image generated according to a camera matrix P corresponding to the viewing angle of the acquired 2D image.

[0117] In an optional step S20, in order to allow accurate determination of a plurality of camera matrices P corresponding to the acquired 2D images (see step S30) and to allow projection of the acquired 3D imaging data according to the plurality of camera matrices P, 3D imaging data is acquired with a calibration device attached to the anatomical specimen. In a sub-step S22 of step S20, before acquiring 3D imaging data and 2D images of a specific body part of the anatomical specimen, the anatomical specimen is provided with a number n of calibration devices. In a sub-step S24, 3D imaging data (referred to as acquired 3D imaging data) of a specific body part of one of the plurality of anatomical specimens is acquired with the calibration device attached. In a sub-step S26, in order to compensate for eventual alignment errors, a 3D-3D registration of the 3D imaging data acquired before the anatomical specimen was provided with the calibration device and the 3D imaging data acquired after the anatomical specimen was provided with the calibration device is performed by the computing device 10. After 3D-3D registration of the 3D imaging data acquired before the anatomical specimen was provided with the calibration device and the 3D imaging data acquired after the anatomical specimen was provided with the calibration device, a synthesized 2D image is generated (in step S54) by projecting the 3D imaging data acquired before the anatomical specimen was provided with the calibration device according to the plurality of camera matrices P. In this manner, the calibration device is not present in the synthesized 2D image (and only the anatomical specimen is present).

[0118] In an optional sub-step S34 of step S30, if the 2D image has been acquired with a calibration device attached, in order to remove the calibration device captured in the 2D imaging data, according to some embodiments, the projection of the calibration device in the 2D image is repaired / removed from the 2D image.

[0119] Figure 2The sub-steps of camera matrix P determination S40 are shown. In a first sub-step S41, the 2D coordinates of the calibration device are detected on the acquired 2D image (referred to as detected 2D coordinates), for example, using a circular Hough transform algorithm applied to the thresholded 2D image. In a sub-step S42, the 3D coordinates of the calibration device are determined based on the 3D imaging data (referred to as detected 3D coordinates), for example, by initial thresholding. Thereafter, in a sequence of repetitive sub-steps S43 to S46, all possible correspondences between 2D coordinates and 3D coordinates (referred to as 2D-3D coordinate correspondences) and the corresponding camera matrices P are determined, the 3D coordinates are projected, and the reprojection error is calculated for each possible correspondence between the 2D coordinates and the 3D coordinates. In the first repetitive sub-step S43, the (next) possible 2D-3D coordinate correspondence of the calibration device is selected. Subsequently, in a sub-step S44, the camera matrix P corresponding to the acquired 2D image is determined based on the currently selected 2D-3D coordinate correspondence, for example, using a direct linear transformation algorithm. After determining the camera matrix P for the currently selected 2D-3D coordinate correspondence, in sub-step S45, the 3D imaging data (captured after setting up the calibration device) are projected according to the plurality of camera matrices P. The projected 3D imaging data are superimposed with the corresponding acquired 2D image, thereby calculating the so-called reprojection error - sub-step S46. The sequence of sub-steps S43 to S46 is repeated for all possible correspondences between 2D coordinates and 3D coordinates. The number of possible correspondences between 2D coordinates and 3D coordinates is generated by the number n of calibration devices captured by both 2D and 3D images, according to the combination formula:

[0120]

[0121] After the reprojection errors have been calculated for all possible correspondences between 2D coordinates and 3D coordinates, the 2D-3D coordinate correspondence with the smallest reprojection error is selected as the correct one - sub-step S47. This 2D-3D coordinate correspondence is then used to determine the camera matrix P for each of the acquired 2D images.

[0122] According to some embodiments, in which a relatively large number of calibration devices are provided in order to enable a more accurate determination of the camera matrix P, only a first subset n′ of said calibration devices (called reference calibration devices) is considered in the sequence of sub-steps S43 to S47 in order to limit the number of all possible correspondences between 2D coordinates and 3D coordinates. In this case, the number n′, which defines the number of all possible correspondences, is the number of reference calibration devices.

[0123] Thereafter, after the 2D-3D coordinate correspondences of the reference calibration device have been determined, in an optional sub-step S48, all calibration devices (not only the reference calibration device) are projected according to the selected plurality of camera matrices P, and estimated 2D coordinates of all calibration devices are determined from these projections. Based on a comparison of these estimated 2D coordinates with the detected 2D coordinates of sub-step S41, the 2D-3D coordinate correspondences of all calibration devices are determined, thereby allowing the camera matrix P to be recalculated using all calibration devices (in sub-step S49). Thus, an optimal balance can be achieved between fitting a large number of calibration devices - allowing an accurate determination of the camera matrix P - and the computational power required to determine the reprojection error of each camera matrix P under all possible 2D-3D correspondences.

[0124] exist Figure 3 As previously shown in Figure 1 、 Figure 2 and Figure 4 The data collection and processing flow for the specific application of fresh-frozen ex vivo whole-torso specimens is described in [ 15 ]. The process begins—in step S12 —by acquiring a plurality of CT scans of fresh-frozen ex vivo whole-torso specimens (e.g., slice thickness: 0.75 mm, in-plane resolution: 0.5 mm×0.5 mm), based on which segmentation of individual body parts (such as vertebrae) in a 3D volume is performed—in step S14.

[0125] Thereafter, in step S22, stainless steel spherical fiducials (a specific embodiment of the calibration device) are placed in the frozen soft tissue in the following manner: first, a Kirschner wire (3 mm diameter) indicating the intended drilling trajectory is inserted, and the intended trajectory is visually verified using fluoroscopic guidance, and if confirmed, subsequent drilling is performed to create a narrow channel in the frozen tissue for placing the fiducials. After the fiducial insertion stage, a super glue with low viscosity can be used to seal the residual channel left in the frozen soft tissue. When placing the fiducials, it should be ensured that the bone structure remains intact and that the fiducials are inserted into an imaginary cylinder that encapsulates the spine and covers the area from L1 to L5. This is to ensure that the fiducials have sufficient separation and are visible at clinically relevant imaging angles (anteroposterior, lateral and oblique).

[0126] Thereafter, in step S24, a CT image after fiducial placement is acquired from the anatomical specimen that explicitly shows the relative placement of the fiducials with respect to the anatomical structures. 3D-3D registration is then performed between the initial CT scan (fixed image) and the CT scan after fiducial placement in step S26, followed by extraction of the 3D fiducial coordinates Xi∈R from this CT image in step S41 by initial thresholding for metallic objects and later performing 3D connected component analysis.3 , a list of i∈[1, 2, …, 14].

[0127] In another step S32, X-ray images of the anatomical specimen are collected, for example, using a clinical-grade mobile C-arm device, starting with the anteroposterior (AP) view as the base viewing angle. Starting from the AP view, the specimen is imaged in 3° increments by rotating the C-arm gantry in the transverse plane until a ±102° orbital angle is reached in both directions. In addition, starting from the AP viewing angle, tilt movement of the C-arm is used in 3° increments to image the specimen in the sagittal plane until a ±25° tilt angle is reached in both directions, thereby producing oblique X-rays. Similarly and starting from the lateral viewing angle, tilt movement is used in 5° increments in the sagittal plane until a ±15° tilt is reached. During the image acquisition phase, the C-arm gantry is positioned so that the X-ray is centered at L3 while capturing as much of the lumbar region as possible.

[0128] Once the X-ray has been acquired, a calibration process is used - step S43 - to calibrate the X-ray using the coordinates of the projected fiducial markers in 2D and their corresponding 3D coordinates (see below). Figure 4 details of the calibration process are described).

[0129] The calculated camera matrix P is decomposed into extrinsic parameters and intrinsic parameters by the following formula:

[0130] P=[KR|-KRX o ]→M=KR,m=-KRX o

[0131] X o =-M -1 m

[0132] qr(M -1 )=R T K -1 →q=R T , r=K -1

[0133] Among them, K is the matrix containing the intrinsic camera parameters, R is the rotation matrix, Xo is the focal coordinate, and qr is used to decompose a matrix into an orthogonal matrix q and an upper triangular matrix r.

[0134] Then—in sub-step S52 of step S50—using the restored intrinsic camera parameters and extrinsic camera parameters, a DRR image I is generated using the restored intrinsic camera parameters and extrinsic camera parameters using the following formula:

[0135] I(p)=∫A(T -1 L(p,s))ds

[0136] Where l(p) is the intensity of DRR at image point p, T = [R|Xo]: R 3 →R 3 is the transformation between the object and the image plane, and L(p,s) is the ray parameterized by s that originates from the X-ray source and intersects the image plane at point p. According to this method, the DRR image is projected from the same viewing point as the acquired X-ray image to obtain paired synthetic-real X-ray data.

[0137] The entire above process (i.e., preoperative CT, fiducial placement, post-fiducial CT, calibration, and downstream DRR generation) is performed for all X-ray images collected from the anatomical specimen. Using the 3D segmentation mask, a 2D mask on the DRR image is generated that is used to localize the synthetic-real X-ray image pairs so that only a single vertebral segment is present on a given image.

[0138] like Figure 4 As illustrated, for Figure 3 The camera matrix P for a specific application is determined as follows:

[0139] -Benchmark detection—sub-step S41:

[0140] First, the 2D coordinates xj∈R of the projected fiducial are detected using the circular Hough transform algorithm applied to the thresholded image 2 , j∈[1, 2, ..., 14]. This can be displayed to the user as a suggestion ( Figure 4 a), the user can then use keyboard and mouse input to accept, reject, or modify the detected points.

[0141] - Calculate 2D-3D coordinate correspondence - sub-steps S43 to S47:

[0142] Up to this stage, the 2D and 3D reference coordinates have been extracted, however the correspondence between the two coordinate lists x∈R2 and X∈R3 has not yet been resolved. To this end, beads with a larger radius are considered and their 3D coordinates as reference references, for which the corresponding 2D coordinates are expected in a first step. Given a given 2D-3D correspondence, the camera matrix P (containing intrinsic and extrinsic imaging parameters) and the associated reprojection error can be calculated using the direct linear transformation (DLT) algorithm. This reprojection error is then used as a cost function in a random sample consensus (RANSAC) framework, where the calculation of the P matrix and the resulting reprojection error are repeated for all possible 2D-3D correspondences of the reference reference. The specific correspondence that produces the minimum reprojection error is considered correct and is used to calculate the camera matrix P, which is a first estimate of the camera matrix P (calculated using only the reference reference) - sub-step S47. This camera matrix P is then used to reproject all reference points from the 3D space X∈R 3 Projection to 2D image space: x = PX, where x∈R 2 is the estimated 2D coordinate of the projected fiducial ( Figure 4 b).

[0143] - Reference 2D-3D coordinate correspondence correction - sub-step S48:

[0144] The benchmark projection estimated by comparison and the detected fiducial projection x, restoring the 2D-3D coordinate correspondence of all fiducials, thereby allowing the (re)calculation of the camera matrix P - sub-step S49 - using all fiducial markers ( Figure 4 c).

[0145] -Baseline repair - step S34:

[0146] The projection of the fiducial markers is then fixed on the acquired X-ray ( Figure 4 d).

[0147] Figure 5 A flowchart showing the steps of a method for - applying a transfer learning process - processing anatomical imaging data, including generating a training dataset, pre-training an artificial intelligence algorithm using the pre-training dataset, training the pre-trained artificial intelligence algorithm using training data from the training dataset, and processing the anatomical imaging data using the trained artificial intelligence algorithm.

[0148] In step S60, the artificial intelligence algorithm is pre-trained using a pre-training dataset. In a first sub-step S62, the pre-training dataset is acquired. In order to overcome the limitations of the availability and / or accuracy of annotated imaging datasets capturing body parts corresponding to specific body parts, the pre-training dataset includes a plurality of imaging datasets generated based on 3D imaging data (particularly computed tomography CT scans) capturing the specific body parts. In particular, different camera matrices P are used to generate - in particular taking into account that the input pre-operative CT scans are generated from different perspectives - synthetic 2D images, such as fluoroscopic images (i.e., DRRs). For example, using this approach, up to hundreds of "synthetic" 2D images (which capture a specific body part) can be generated based on a single CT scan. In a subsequent sub-step S64 of step S60, the artificial intelligence algorithm is pre-trained using the pre-training dataset.

[0149] Thereafter, in step S70, using the Figures 1 to 4 A pre-trained artificial intelligence algorithm is (re)trained using a training dataset generated by one of some embodiments disclosed herein (described herein). Training the pre-trained artificial intelligence algorithm (also referred to as retraining) includes fixing a first subset of network parameters of network components of the pre-trained artificial intelligence algorithm (such as weights and biases of an artificial neural network), and adjusting a second subset of network parameters using training data pairs of the training dataset as input. For one specific embodiment of reconstructing the anatomical 3D shape of a specific body part of an anatomical object, retraining the pre-trained artificial intelligence algorithm includes fixing network parameters of network components (except a 2D feature extractor module) of the pre-trained artificial intelligence algorithm (such as weights and biases of an artificial neural network), and adjusting the network parameters of the 2D feature extractor module using training data pairs of the training dataset as input.

[0150] As a next main step S80 , a 2D image of a specific body part of an anatomical object, such as a patient during surgery or medical examination, is acquired.

[0151] Thereafter, in step S100 , the acquired 2D image is processed using an artificial intelligence algorithm that is first pre-trained using the pre-training dataset and then (re-)trained using the training dataset.

[0152] In the first (alternative or additional) sub-step S102 of step S100, processing the acquired 2D image of the specific body part of the anatomical object using an artificial intelligence algorithm includes: reconstructing the anatomical 3D shape of the specific body part of the anatomical object based on the acquired 2D image of the specific body part and data indicating a camera matrix P (in particular, the viewing angle corresponding to the acquired 2D image).

[0153] As a second (alternative or additional) sub-step S104 , processing the acquired 2D image of the specific body part of the anatomical object using the artificial intelligence algorithm includes: performing anatomical segmentation on the acquired 2D image to identify one or more anatomical characteristics of the specific body part.

[0154] Figure 6 A specific embodiment of retraining a pre-trained artificial intelligence algorithm is shown. Considering that the artificial intelligence algorithm has been pre-trained (and tested) using synthetic 2D images, the purpose of retraining is to use data pairs (acquired 2D images and corresponding synthetic 2D images of the training dataset generated according to one of some embodiments disclosed herein) together with the restored camera matrix P (i.e., intrinsic imaging parameters and extrinsic imaging parameters) to retrain the pre-trained artificial intelligence algorithm in a transfer learning manner.

[0155] like Figure 6 As shown above, the pre-trained artificial intelligence algorithm accepts a localized 2D image along with its corresponding imaging parameters as its input. As the first step of the retraining process, a UNet model is used to extract a 2D feature map to be later passed to a differentiable back-projection module, which creates a 3D feature grid using the input imaging parameters (i.e., the camera matrix P). Later, the 3D feature grid is averaged and input to a refiner UNet model, which in turn outputs a reconstructed 3D shape. During the transfer learning process, the pre-trained artificial intelligence algorithm is retrained by fixing the network parameters (weights and biases) for all network components except the network parameters of the 2D feature extractor module. During the transfer learning process, the trainable network parameters for the 2D feature extractor module are adjusted using the collected training data pairs as input.

[0156] Figure 7 A flowchart is shown illustrating the steps of a method for processing anatomical imaging data by applying a style transfer process, including generating a training dataset, pre-training an artificial intelligence algorithm using the pre-training dataset, training a style transfer algorithm using the training dataset, pre-processing the anatomical imaging data using the style transfer algorithm by applying style transfer to acquired 2D images, and processing the pre-processed anatomical imaging data using the pre-trained artificial intelligence algorithm.

[0157] Alternative Figure 5 and Figure 6 The transfer learning method described in Figure 7 and Figure 8 This method of the embodiment shown above. In which the setting of the domain adaptation framework is solved by style transfer processing on the acquired (real) 2D images, using artificial intelligence algorithms pre-trained on synthetic images. Figure 5Similar to the description of transfer learning shown in , step S50 - generating a training dataset - and step S60 - pre-training the artificial intelligence algorithm using the pre-training dataset are performed. Then, instead of performing Figure 5 Instead of the transfer learning in step S70 in step S65, the style transfer learning algorithm is trained using the training dataset. Similar to the transfer learning, a 2D image of the anatomical object is captured in step 70. Figure 5 Unlike the transfer learning shown in , the trained transfer learning algorithm is then applied to the acquired (or captured) 2D image in step S90. Similar to transfer learning, the anatomical 2D image is processed in step S100, although in this case - unlike Figure 5 Compared to using a captured 2D image that has already been processed by a style transfer algorithm. In other words, rather than performing transfer learning (step S70), a style transfer algorithm is trained (S65) and applied to the acquired 2D image (S90). Thus, processing the anatomical 2D image (S100) is performed on the pre-processed / enhanced 2D image.

[0158] Specifically, in step S65 , a style transfer algorithm is trained using the training dataset. The style transfer algorithm is trained using the training dataset to extract texture information from the synthesized 2D image and transfer the texture information to the acquired 2D image while preserving the basic semantic content of the acquired 2D image.

[0159] After the style transfer algorithm has been trained, in step S90 , the style transfer algorithm is applied to the acquired 2D image (referred to as pre-processed acquired 2D image) before processing the acquired 2D image (using the artificial intelligence algorithm).

[0160] exist Figure 8 Step S90 of applying the style transfer algorithm is shown in more detail in FIG. In sub-step S92 of the style transfer process, texture information is extracted from the synthesized 2D image of the training dataset. In sub-step S94, the pre-processed acquired 2D image of the anatomical object is synthesized by transferring this texture information onto the acquired 2D image while preserving the essential semantic content of the acquired 2D image.

[0161] Known style transfer methods for intraoperative 2D imaging data typically deal with the migration from synthetic 2D imaging (i.e., DRR) to acquired 2D imaging (i.e., X-ray), hoping to achieve more realistic synthetic 2D images for training artificial intelligence algorithms. In contrast, the key idea of ​​the present style transfer method is to apply the style transfer concept in the opposite direction, to migrate acquired (i.e., X-ray) 2D images to the synthetic 2D imaging (i.e., DRR) domain, for the following reasons:

[0162] -Even under controlled imaging conditions, acquired (i.e., X-ray) 2D images have large variations in image and radiological properties, depending on factors such as: imaging technique, detector type (intensifier vs. flat panel), equipment settings, patient's body mass index (BMI), imaging viewing angle, and backscatter. This makes the task of synthetic to acquired 2D imaging (i.e., DRR to X-ray) migration inherently challenging, as the target domain itself (i.e., X-ray domain) has high heterogeneity in this setting. In contrast, according to the present invention, a synthetic (i.e., DRR) 2D image is created using a constant threshold applied to the underlying acquired 3D image (i.e., CT scan). The effect is that, for example, all CT voxels with a Hounsfield value below a threshold are ignored during the DRR generation phase. This threshold is chosen empirically to project only bone structures on the synthetic (i.e., DRR) 2D image, which results in a homogeneous target domain.

[0163] -Similar to other AI-based methods that use synthetic (i.e., DRR) 2D image data for training purposes, the pre-trained AI algorithm of the present invention has been pre-trained using a large number of annotated synthetic (i.e., DRR) 2D images. Therefore, by providing an acquired (i.e., X-ray) to synthetic (i.e., DRR) style transfer algorithm, the acquired (i.e., X-ray) 2D images can be first converted to the synthetic (i.e., DRR) domain and then the pre-trained AI algorithm can be used.

[0164] Using the training dataset (generated according to the method of the present invention), four different types of style transfer algorithms can be applied according to various embodiments.

[0165] - Encoder-Decoder Deep Convolutional Network (EDDC): This style transfer algorithm can be implemented using an hourglass encoder-decoder architecture. To train the EDDC network, a training dataset generated according to any of the embodiments of the present invention, comprising localized paired synthetic-real 2D images, is used in the following way: an acquired (i.e., X-ray) 2D image is considered as the network input, and the corresponding localized synthetic (i.e., DRR) image is considered as the network target.

[0166] Neural Style Transfer (NST): This algorithm separates the style and content of an input image by using a deep convolutional network as a feature extractor. To generate the output image, content information from the acquired (i.e., X-ray) image is extracted and combined with style information extracted from a synthesized (i.e., DRR) image. The core of this algorithm are two specialized loss functions, one tasked with estimating the content similarity (i.e., content loss) between the generated image and the input acquired (i.e., X-ray) image, and the other tasked with estimating the style similarity between the generated image and the input synthesized (i.e., DRR) image. To train NST, a localized paired synthetic-real training dataset generated according to any of the embodiments of the present invention is used.

[0167] - Deep Convolutional Generative Adversarial Network (DCGAN): Generative Adversarial Networks have been widely used for image synthesis tasks, when GAN models are trained to generate new images based on random noise input. Figure 9 As shown in

[15] , the GAN-based image synthesis task is converted into a style transfer task by exchanging random noise input with real / acquired (i.e., X-ray) 2D images. This can be achieved with the help of a localized paired synthetic-real training dataset generated according to any of the embodiments of the present invention.

[0168] - Cycle-Consistent Generative Adversarial Networks (CycleGAN): In cases where only unpaired training data is available for style transfer purposes, a CycleGAN network is preferred, which ensures reversible image-to-image transfer and includes mechanisms for preventing mode collapse. To address this use case, one can use Figure 10 The CycleGAN network architecture shown in , which includes two generators and two discriminator networks. Considering that unpaired training data is sufficient to train such a CycleGAN network, a localized paired synthetic-real training dataset (generated according to any of the embodiments of the present invention) and an additional larger synthetic (i.e., DRR) 2D imaging data dataset can be used. This can be performed to identify whether only unpaired synthetic-real data is sufficient for the desired style transfer task.

[0169] Now go to Figure 11 to Figure 1 9. Specific practical applications of a method for generating training data and a method for training an artificial intelligence algorithm that assists in positioning a tool (such as a surgical tool) relative to a specific body part of a patient (reconstructing the anatomical 3D shape (AS) of the specific body part based on intraoperative 2D imaging data (ID) and data indicating the viewing angles corresponding to multiple intraoperative 2D images) will be described.

[0170] Figure 11 A highly schematic stereoscopic view of a system 1 for assisting in positioning a tool 5 as installed in an operating room is shown, with a patient 200 lying on an operating table 2. As illustrated, the system 1 includes a computing device 10; an intraoperative imaging device 20; and a display device 30. The system 1 is illustrated based on an embodiment utilizing radiation-based images as intraoperative images. Therefore, the intraoperative imaging device 20 includes a C-arm intraoperative imaging device 20 that captures intraoperative images using X-ray technology. The C-arm intraoperative imaging device 20 includes a generator (X-ray source) 22. The C-shaped connecting element (C-arm) 24 allows movement horizontally, vertically and / or around a rotation axis, so that 2D X-ray images of the patient 200 can be generated from multiple viewing angles around the patient. The generator 22 emits X-rays that penetrate the patient's body 200. The detector 26 converts the X-rays into imaging data ID that is transmitted to the computing device 10.

[0171] The intraoperative imaging device 20 is communicatively connected to the computing device 10 and is disposed in the vicinity of the patient 200, thereby allowing the intraoperative imaging device 20 to capture intraoperative 2D imaging data ID of the patient 200, such that two or more of the intraoperative 2D images capture a specific body part 202 of the patient 200 from two or more different perspectives relative to the specific body part 202 of the patient 200. One or more of the same plurality of intraoperative 2D images capturing the specific body part 202 also capture at least a portion of the tool 5 from at least one perspective.

[0172] In the illustrated embodiment, the display device 30 includes a series of computer screens 32 that are communicatively connected to the computing device 10 and configured to display the guidance data GD.

[0173] Now go to Figure 12 , the steps of a computer-implemented method for assisting in positioning a tool 5 relative to a specific body part 202 of a patient 200 will be described.

[0174] like Figure 12 As shown above, the method includes the following main steps:

[0175] -Step S110: Capturing intraoperative 2D imaging data;

[0176] -Step S120: receiving intraoperative 2D imaging data;

[0177] - Step S130 : Using the intraoperative 2D imaging data and using an artificial intelligence algorithm corresponding to the specific body part 202 , reconstruct the anatomical 3D shape;

[0178] - Step S140: estimating the current position of the tool 5c based on the intraoperative 2D imaging data ID;

[0179] - Step S150: identifying the prescribed position 5p of the tool;

[0180] - step S160: reconstructing positioning guidance data GD comprising a visual representation of the estimated current position 5c of the tool 5 relative to the anatomical 3D shape AS of the specific body part 202; and

[0181] -Step S170: Output the positioning guidance data GD using the display device 30.

[0182] Steps specific to particular embodiments are illustrated in the figures with dashed lines.

[0183] In step S110, intraoperative 2D imaging data ID is captured by the intraoperative imaging device 20 disposed near the patient 200. The intraoperative 2D imaging data ID includes data indicating viewing angles corresponding to a plurality of intraoperative 2D images, the viewing angles identifying the position and / or orientation of the intraoperative imaging device 20 that captured the plurality of intraoperative 2D images, such as the position of the intraoperative imaging device 20 in an x, y, and z Cartesian coordinate system and / or the orientation such as roll, pitch, and heading relative to a specific body part 202 of the patient 200.

[0184] According to a first embodiment, data indicating the viewing angles corresponding to a plurality of intraoperative 2D images is stored in a data storage device included in or communicatively connected to the computing device 10. The viewing angles stored in the data storage device are determined by tracking the C-arm 24 to estimate imaging parameters at the time of exposure, by which the intraoperative 2D images acquired during the operation can be assigned to their respective intrinsic and extrinsic imaging parameters, which effectively define the viewing angles from which the intraoperative 2D images have been generated. Optionally, the tracking of the C-arm 24 is preceded by a calibration process, namely a preoperative calibration process (i.e., precalibration), in which the C-arm 24 is manipulated in a specific manner to cover the expected range of motion. During this precalibration phase, a mathematical relationship between the tracking observations and the imaging parameters is established at specific pose intervals, which will later be used to derive an interpolation function that can generate the intraoperative imaging parameters based on the tracking data.

[0185] Alternatively or additionally, the viewing angle corresponding to the intraoperative 2D imaging data ID is estimated based on the intraoperative 2D imaging data ID by the computing device 10. In one embodiment, the calibration algorithm extracts the viewing angle of the intraoperative 2D image by placing a precisely manufactured calibration object (i.e., a phantom) (the calibration object includes different features (e.g., radiopaque features) with known geometry) in the imaging field and estimating imaging parameters based on projections of these features.

[0186] Alternatively or additionally, an artificial intelligence algorithm is used to estimate the viewing angle corresponding to the intraoperative 2D imaging data ID, and the artificial intelligence algorithm is trained using a large number of imaging data sets with known viewing angles. In order to overcome the limitations of the availability and / or accuracy of imaging data sets with known viewing angles, a large number of imaging data sets including intraoperative 2D images from known viewing angles are generated based on 3D imaging data (particularly computed tomography CT scans). For example, a convolutional neural network (CNN) for regression tasks is trained using simulated intraoperative fluoroscopic photographs (i.e., digitally reconstructed radiographs (DRRs)) generated based on preoperative CT scans, together with their corresponding posture parameters. Using this artificial intelligence algorithm trained prior to the surgical procedure, the intraoperative positioning of the intraoperative imaging device 20 can be estimated based solely on the intraoperative images, without the need for external tracking devices or calibration phantoms.

[0187] In subsequent step S120 , the intraoperative 2D imaging data ID is received by the computing device 10 from the intraoperative imaging device 20 through its data input interface 14 .

[0188] In subsequent step S130, the computing device 10 reconstructs the anatomical 3D shape AS of the specific body part 202 based on the intraoperative 2D imaging data ID and the data indicating the viewing angles corresponding to the plurality of intraoperative 2D images using an artificial intelligence algorithm corresponding to the specific body part 202. Figure 13 、 Figure 14 、 Figure 15 、 Figure 16 and Figure 17 A detailed description of step S130 of reconstructing the anatomical 3D shape AS is provided.

[0189] In step S140, the current position 5c of the tool 5 relative to the anatomical 3D shape AS of the specific body part 202 is estimated based on the intraoperative 2D image of the imaging data ID capturing the tool 5. The current position 5c of the tool 5 is determined based on a priori knowledge of the geometry of the tool 5 as described by the tool geometry model. First, a projection of the tool geometry model is compared with at least a portion of the tool 5 as captured by the corresponding 2D image of the intraoperative 2D imaging data ID. The tool geometry model is projected onto one or more planes in the intraoperative 2D image of the intraoperative 2D imaging data ID capturing at least a portion of the tool 5. The planes of the intraoperative 2D images of the intraoperative 2D imaging data ID are determined based on the viewing angle of each 2D image. Thereafter, a position of the tool geometry model projected onto the plane of the intraoperative 2D image of the intraoperative 2D imaging data ID is determined that (best) matches at least a portion of the tool 5 as captured by the corresponding 2D image of the intraoperative 2D imaging data ID.

[0190] According to some embodiments disclosed herein, although the anatomical 3D shape is reconstructed once in an initial stage of the method for assisting in positioning a tool, the estimation of the current position 5c of the tool 5 is repeatedly performed at set intervals and / or triggered by certain events and / or manually triggered.

[0191] To improve the accuracy of the estimated positioning of tool 5, tool 5 is verified based on a tool geometric model. The tool geometric model is specifically designed to optimize the estimation of its positioning based on as few intraoperative 2D images as possible. In particular, the tool is designed so that at least a portion of it is not completely rotationally symmetric about any axis of the Cartesian coordinate system, allowing the tool's orientation to be estimated based on the intraoperative 2D images. Alternatively or additionally, the tool is designed to include special markings to facilitate its identification based on the 2D intraoperative images.

[0192] In step S150, the prescribed position 5p of the tool 5 relative to the anatomical 3D shape AS of the specific body part 202 is identified by the computing device 10, and the visual representation of the prescribed position 5p of the tool 5 is superimposed on the visual representation of the estimated current position 5c of the tool 5 to assist the surgeon in correctly positioning the tool 5.

[0193] refer to Figure 18 An embodiment of determining a prescribed position 5p of the tool 5 is described.

[0194] Figure 13 A flowchart is shown that illustrates the steps of reconstructing an anatomical 3D shape AS based on intraoperative 2D imaging data ID and data indicating viewing angles corresponding to a plurality of intraoperative 2D images. As illustrated, the reconstruction of the anatomical 3D shape AS is performed in two stages: step S132 - segmenting the intraoperative 2D imaging data ID to identify a specific body part 202 of the patient 200; and step S134 - further reconstructing the anatomical 3D shape AS using the segmented intraoperative 2D imaging data ID. Step S132 - segmenting the intraoperative 2D imaging data ID using an artificial intelligence-based detection and segmentation model to identify a specific body part 202 of the patient 200 is illustrated in FIG. Figure 14 In the above, it is applied to segment intraoperative 2D images of the spine to identify individual vertebrae. In order to train the AI-based detection and segmentation model, given the input preoperative CT scan, synthetic X-rays (i.e., DRRs) are generated based on different viewpoints around the patient 200. The CT scans used for this purpose can be collected through a public dataset that includes CT scans together with corresponding vertebral segment (level) annotations. For example, using this method, a training database with more than 40,000 annotated intraoperative 2D images can be created based on only 200 preoperative CT scans.

[0195] Figure 15 A schematic illustration of another embodiment of step S132 for segmenting intraoperative 2D imaging data ID according to a two-stage approach is shown, including identifying a region of interest, followed by semantic segmentation of the region of interest to identify a specific body part 202 of the patient 200 within the region of interest. To segment the intraoperative 2D imaging data ID, an artificial intelligence-based detection and segmentation model (such as a convolutional neural network-based detection and segmentation model) is first used to identify a region of interest within the intraoperative 2D imaging data ID, the region of interest containing the specific body part 202 of the patient 200. The artificial intelligence-based detection and segmentation model is then used to semantically segment the region of interest, thereby generating segmented intraoperative 2D imaging data ID. The artificial intelligence-based detection and segmentation model used for the segmentation in step S132 is trained using supervised learning. First, a convolutional neural network (CNN)-based detection model is trained to identify individual body parts (vertebral segments in the illustrated embodiment) on the intraoperative 2D image by detecting the coordinates of bounding boxes each encompassing a single body part (a single vertebra). The identified bounding boxes are then used as regions of interest to crop the intraoperative 2D image. Furthermore, an end-to-end segmentation model is trained to semantically segment the projections of the vertebrae within the region of interest. During the inference phase, intraoperative X-ray images are fed into the segmentation model, which produces semantic segmentations for each vertebral segment (to be used for 3D reconstruction purposes).

[0196] Figure 16 shows the intraoperative 2D imaging data ID based on the segmentation and the viewing angle P corresponding to the plurality of intraoperative 2D images. 1-n Schematic illustration of reconstructing the anatomical 3D shape AS of a specific body part 202 using an artificial intelligence algorithm corresponding to the specific body part 202 based on the data of the intraoperative 2D imaging data ID. As illustrated, the segmented intraoperative 2D imaging data ID is back-projected into a 3D coordinate system to create an anatomical 3D shape for each body part 202 (in this case, a vertebra). The back-projection is based on the view angle P of the intraoperative 2D image. 1-n Each 2D image provides information about the body part 202 from its perspective. Thus, the anatomical 3D shape AS is incrementally constructed, and the more intraoperative 2D images the intraoperative data ID includes, the more accurate the reconstructed anatomical 3D shape AS becomes - as shown in FIG. Figure 16 The lower part is illustrated in the sequence of anatomical 3D shapes.

[0197] In addition to the reference Figure 13 In addition to steps S132 and S134 described above, Figure 17In some other embodiments illustrated in the flowchart of FIG. 1 , in another step S136 , the non-segmented imaging data is used to further enhance the anatomical 3D shape AS init Taking into account the potential errors in calibration and segmentation, a 3D shape enhancement model is used to enhance the reconstructed initial anatomical 3D shape AS init The 3D shape enhancement model (specifically the Convolutional Neural Network (CNN) architecture) is provided with two input streams. The first input stream consists of the anatomical 3D shape AS init The second input stream includes a 2D segmentation of a specific body part 202 on the intraoperative 2D image of the intraoperative 2D imaging data ID. In this way, by injecting the patient-specific shape information retained in the original intraoperative 2D image, the 3D shape enhancement model is trained to complete the missing parts of the initial reconstruction (taking into account the possibility of data loss during the initial reconstruction due to missing projection views), thereby reconstructing an enhanced anatomical 3D shape AS enh .

[0198] Now go to Figure 18 , one embodiment of determining a prescribed position 5p of a tool 5 is described with reference to a use case of a surgical procedure of implanting a pedicle screw into a vertebra of a patient 200. The prescribed position 5p of the tool 5 is determined by an artificial intelligence-based optimization function based on the anatomical 3D shape AS of the specific body part 202 and data indicative of the surgical procedure.

[0199] Determining the prescribed position 5p of the tool 5 based on the anatomical 3D shape AS is advantageous because it has the potential to improve surgical workflow by eliminating the need for preoperative scanning and the corresponding manual process, which can be both expensive and time-consuming. An artificial intelligence-based optimization function is trained using supervised learning and reinforcement learning (RL) based on a clinical dataset containing ideal screw trajectories identified by experts. The prescribed position 5p of the tool 5 is then determined based on the ideal screw trajectory IST and further on prior knowledge of the geometry of the tool 5.

[0200] Figure 19A 、 Figure 19B and Figure 19C Some implementations of the positioning guidance data GD are shown. Figure 19A Positioning guidance data GD is shown comprising a visual representation of an estimated current position 5c of the tool 5 and a visual representation of a prescribed position 5p of the tool 5 superimposed onto a visual representation of the reconstructed anatomical 3D shape AS.

[0201] Figure 19BPositioning guidance data GD is shown comprising a visual representation of the estimated current position 5c of the tool 5 superimposed onto the 2D image of the intraoperative 2D imaging data ID.

[0202] Figure 19C Positioning guidance data GD are shown, comprising a visual representation of the estimated current position 5c of the tool 5 onto the visual representation of the reconstructed anatomical 3D shape AS, a visual representation of the prescribed position 5p of the tool 5 and a visual representation of the ideal screw trajectory of the surgical implant.

[0203] Reference Signs List

[0204] 1 System

[0205] 2 Operating Table

[0206] 5 Tools

[0207] Current Positioning of 5c Tools

[0208] 5p tool’s stated positioning

[0209] 10 Computing Devices

[0210] 12 Data output interface (of computing equipment)

[0211] 14 Data input interface (of computing equipment)

[0212] 16 processing units (of a computing device)

[0213] 18 Storage unit (of a computing device)

[0214] 20 Imaging equipment

[0215] 22 Generator (X-ray source)

[0216] 24 C-shaped connecting element (C-arm)

[0217] 26 detectors

[0218] 30 Display devices

[0219] 32 Computer Screen Arrangement

[0220] 200 patients

[0221] 202 Specific body part (of the patient)

[0222] ID intraoperative imaging data

[0223] GD positioning guidance data

[0224] AS Anatomical 3D Shape

[0225] AS_init Initial reconstruction of anatomical 3D shape

[0226] AS_enh Enhanced anatomical 3D shapes

[0227] IST ideal pedicle screw trajectory

[0228] P 1-n Viewing angle (of 2D intraoperative images)

Claims

1. A computer-implemented method for processing anatomical imaging data, the method comprising, for a plurality of anatomical specimens: - acquiring 3D imaging data of a specific body part of one of the plurality of anatomical specimens; - acquiring 2D images of the specific body part of the anatomical specimen from a plurality of different viewing angles relative to the specific body part; - Determine a plurality of camera matrices P corresponding to the acquired 2D images, where: The camera matrix P at least indicates the viewing angle of the 2D image relative to the specific body part; - generating a synthesized 2D image by projecting the acquired 3D imaging data according to the plurality of camera matrices P; as well as - associating an acquired 2D image with one of the synthesized 2D images according to the plurality of camera matrices P as a training data pair, The method also includes using the training data pairs to generate a training data set for training an artificial intelligence algorithm for processing anatomical imaging data.

2. The computer-implemented method of claim 1 , further comprising: - providing the anatomical specimen with a calibration device before acquiring the 3D imaging data and the 2D image of the specific body part of the anatomical specimen; - determining the 2D coordinates of the calibration device in each of the 2D images; - determining the 3D coordinates of the calibration device based on the 3D imaging data; - determining a 2D-3D coordinate correspondence between the 2D coordinates of the calibration device and the 3D coordinates of the calibration device in each of the 2D images; - further using the 2D-3D coordinate correspondence to determine the plurality of camera matrices P corresponding to the acquired 2D images.

3. The computer-implemented method of claim 2 , further comprising: - before arranging the anatomical specimen with the calibration device, acquiring 3D imaging data capturing the specific body part of the anatomical specimen; - performing 3D-3D registration of the 3D imaging data acquired before the anatomical specimen is provided with the calibration device and the 3D imaging data acquired after the anatomical specimen is provided with the calibration device; - generating the synthesized 2D image by projecting the 3D imaging data acquired before the anatomical specimen is provided with the calibration device according to the plurality of camera matrices P; as well as - inpainting the projection of the calibration device from the acquired 2D images before the acquired 2D images are used to generate the training dataset.

4. The computer-implemented method of claim 2 or 3, wherein: Determining the plurality of camera matrices P corresponding to the acquired 2D images comprises: - detecting the 2D coordinates of the calibration device on the acquired 2D image; - detecting the 3D coordinates of the calibration device on the acquired 3D imaging data; - determining a 2D-3D coordinate correspondence between the 2D coordinates and the 3D coordinates of the calibration device; and - determining the camera matrix P using the 2D coordinates and the 3D coordinates of the calibration device and the 2D-3D coordinate correspondence.

5. The computer-implemented method according to any one of claims 1 to 4, further comprising: - obtaining a pre-training dataset comprising a plurality of synthetic 2D images capturing the specific body part of an anatomical specimen; as well as - Using the pre-training dataset to pre-train an artificial intelligence algorithm for processing anatomical imaging data.

6. The computer-implemented method of claim 5, further comprising training a pre-trained artificial intelligence algorithm for processing anatomical imaging data using the training data set.

7. The computer-implemented method of claim 6, wherein: Training the pre-trained artificial intelligence algorithm includes: - fixing a first subset of network parameters of said pre-trained artificial intelligence algorithm, and - using the training data pairs of the training dataset as input to adjust a second subset of network parameters of the pre-trained artificial intelligence algorithm.

8. The computer-implemented method of claim 5 , further comprising: - using the training dataset to train a style transfer algorithm; as well as - Applying the style transfer process to the acquired 2D image using the style transfer algorithm.

9. The computer-implemented method of claim 8, wherein: The style transfer process includes: - extracting texture information from the synthesized 2D images of the training dataset; and - migrating the texture information to the acquired 2D images of the training data pair while preserving the basic semantic content of the acquired 2D images.

10. The computer-implemented method of any one of claims 5 to 9, further comprising: - acquiring a 2D image of the specific body part of an anatomical subject; as well as - Processing the acquired 2D images of said specific body part of said anatomical object using a trained and / or pre-trained artificial intelligence algorithm.

11. The computer-implemented method of claim 10, wherein: Processing the acquired 2D image of the specific body part of the anatomical object using the artificial intelligence algorithm includes: reconstructing the anatomical 3D shape (AS) of the specific body part of the anatomical object based on the acquired 2D image of the specific body part and data indicating the camera matrix (P) corresponding to the acquired 2D image.

12. The computer-implemented method of claim 10 or 11, wherein: Processing the acquired 2D image of the specific body part of the anatomical object using the artificial intelligence algorithm includes performing anatomical segmentation on the acquired 2D image to identify one or more anatomical characteristics of the specific body part.

13. The computer-implemented method according to any one of claims 5 to 12, further comprising assisting in positioning the tool (5) relative to a specific body part (202) of the patient (200), comprising: a) receiving, by a computing device (10), intraoperative imaging data (ID) from an imaging device (20) disposed in the vicinity of the patient (200), the intraoperative imaging data (ID) comprising intraoperative 2D images, a plurality of intraoperative 2D images capturing the specific body part (202) of the patient (200) from a plurality of different perspectives relative to the specific body part (202) of the patient (200), and one or more of the plurality of intraoperative 2D images capturing at least a portion of the tool (5) from at least one perspective; b) reconstructing, by the computing device (10), an anatomical 3D shape (AS) of the specific body part (202) using a trained artificial intelligence algorithm corresponding to the specific body part (202) based on the intraoperative imaging data (ID) and data indicating viewing angles corresponding to the plurality of intraoperative 2D images; c) estimating, by the computing device (10), a current positioning (5c) of the tool (5) relative to the anatomical 3D shape (AS) of the specific body part (202) based on the intraoperative imaging data (ID); as well as d) generating, by the computing device (10), positioning guidance data (GD), the positioning guidance data (GD) comprising a visual representation of the estimated current positioning (5c) of the tool (5) relative to the anatomical 3D shape (AS) of the specific body part (202).

14. A computing device (10) comprising a processing unit (16) configured to perform the method according to one of claims 1 to 13.

15. A system (1), comprising: - A computing device (10) according to claim 14; - an imaging device (20), such as a pre-operative, post-operative and / or intra-operative imaging device, communicatively connected to the computing device (10) and configured to capture 2D images of a specific body part of an anatomical object.

16. A computer program product comprising instructions which, when executed by a processing unit (16) of a computing device (10), cause the computing device (10) to perform the method according to any one of claims 1 to 13.

Images