High resolution bone models from low resolution pre-operative MRI scans

Abstract

Imaging methods and apparatus, a method comprising: training a neural network (NN) with inputs that include training low-resolution (LR) magnetic resonance imaging (MRI) scans; and generating a high-resolution (HR) output directly from an input LR MRI scan selected from the group consisting of a training input LR MRI scan and a new input LR MRI scan using the trained NN, wherein a spacing of the HR output is smaller by at least 1mm along at least one axis compared to a spacing in the input LR MRI scan along the respective at least one axis.

Description

[0001] HIGH RESOLUTION BONE MODELS FROM LOW RESOLUTION PREOPERATIVE MRI SCANS

[0002] FIELD

[0003] The subject matter disclosed herein relates in general to medical imaging and in particular to using pre-operative low-resolution magnetic resonance imaging (MRI) scans to perform preoperative (for brevity “pre-op”) procedure planning and intra-operative registration and navigation in orthopaedic procedures.

[0004] BACKGROUND

[0005] Modern orthopaedic surgery, such as spinal surgery, is becoming increasingly advanced and precise due to recent technological advances such as 3-dimensional (3D) printing of implants and robotic surgery. A prerequisite for such surgery is an accurate pre-op 3D model of the bone structure of a patient being operated on. Such a model is needed both for preoperative planning as well as for registration to intra-operative images and navigation during the procedure.

[0006] Currently, a known state of the art method for obtaining a pre-op 3D model, shown in FIG. 1, includes step 102 of providing a high-resolution computerized tomography (CT) scan and step 104 of creating a high-resolution (or simply “HR”) bone segmentation mask from the HR CT Scan. In this description, the term “high-resolution” when applied to a 3D image or model is taken to mean that the image contains sufficient detail of the relevant bone structure such that it can be used for pre-op planning and / or intraoperative registration and / or intraoperative navigation without further processing. For example, for typical spinal procedures, CT scans with voxel spacing of 1mm or less in all axes provide sufficient detail of the spine structure. As a general rule, most modern procedures require resolution of 1mm or better, though some less complex procedure may require less resolution. Step 104 can be implemented manually by a human annotator, or by using a computer algorithm. For example, recently highly mature trained neural networks (NNs) have been developed to accurately segment bone structure from CT scans. However, a major disadvantage of this method is that it requires a CT scan to be performed on the patient, thus exposing him / her to a large radiation dosage, which can pose a significant safety hazard. In contrast with CT scans, MRI scans pose no safety hazard to the patient, and may be performed as often as needed. In fact, in many cases MRI scans are used to diagnose the conditions that led to the decision to perform surgery in the first place, so that many patients already have pre-existing MRI scans before surgery.

[0007] Therefore, it would be beneficial to be able to create the pre-op 3D bone model directly from a MRI scan, without performing a CT scan specifically for this purpose. However, MRI scans are generally of lower resolution, meaning that at least one of the axes of the scan has a voxel spacing that is larger by at least 1mm than what is required to provide sufficient detail of the bone structure for the desired procedure. For example, a typical spinal MRI scan may have a spacing as high as 4 or 5 mm along one of its axes (for example, the axial direction normal to the axial plane, or the sagittal direction normal to the sagittal plane), whereas a spacing of 1mm or better is required. FIG. 19 below shows an axial cross section of an exemplary LR MRI scan compared to a corresponding HR MRI scan of the same patient. An additional complication is that even if the spacing of the MRI scan is sufficiently low in all axes, the contrast of part of the bone structure relative to the surrounding tissue may not be sufficient to accurately determine the boundary of the bone structure. For example, the bone cortex in MRI scans typically has insufficient contrast to accurately determine its external boundary.

[0008] To overcome the higher voxel spacing of low-resolution (or simply “LR”) MRI scans, it is possible in theory to up-sample the scan, as for example described in US Patent 11,422,217. However, while such methods may result in an image with lower voxel spacing, the resulting detail of the bone structure relevant to the orthopaedic procedure being performed is insufficient. Furthermore, in cases where the up- sampling method involves supervised training of a NN, it is necessary to generate pairs of LR and HR MRI scans, which is a costly and timeconsuming process. Finally, the up-sampling procedure is typically very sensitive to the MRI protocol used, and therefore may need to be fine tuned for each protocol.

[0009] There is therefore a need for a method to generate a HR pre-op 3D bone model directly from a LR MRI scan. Preferably, the method should be insensitive to the MRI protocol used to generate the scan.

[0010] SUMMARY

[0011] In various exemplary embodiments, there is provided an imaging method, comprising: training a neural network (NN) with inputs that include training low-resolution (LR) magnetic resonance imaging (MRI) scans, and generating a high-resolution (HR) output directly from an input LR MRI scan selected from the group consisting of a training input LR MRI scan and a new input LR MRI scan using the trained NN, wherein a spacing of the HR output is smaller by at least 1mm along at least one axis compared to a spacing in the input LR MRI scan along the respective at least one axis.

[0012] In some examples, the HR output includes an artificial computerized tomography (CT)- like scan. In some examples, the artificial CT-like scan represents soft tissues with a shade or texture that is different to the bone representation.

[0013] In some examples, the HR output includes a segmentation mask. In some examples, a method further comprises generating a HR artificial CT-like scan from the HR segmentation mask.

[0014] In some examples, the training of the NN includes training the NN using CT data that includes CT scans and a HR segmentation mask created from the CT scans. In some examples, the training of the NN includes at least one of: pretraining the NN using the CT scans, training a second NN using the CT data that includes shared weights with the NN, using an autoencoder trained from the CT data to post process the segmentation mask produced by the NN, or using the CT data to pretrain a discriminator, and training the discriminator together with the NN in an adversarial manner.

[0015] In some examples, the generating of the HR output directly from the input LR MRI scan using the trained NN includes acquiring a database of LR MRI scans and associated CT scans belonging to the same patient, training the NN using the LR MRI scans as inputs and using the CT scans to generate ground truth for the HR output. In some examples, the ground truth includes a HR segmentation mask registered to the training input LR MRI scan.

[0016] In some examples, the HR output is used to improve the registration, and the process of registering the ground truth HR segmentation mask to the input LR MRI scan, training the NN using the registered scans, and using the HR output to improve the registration is repeated iteratively. In some examples, the training input LR MRI scan and associated CT scans are spine scans, and the registration is performed separately for each vertebra. In some examples, two or more of the ground truth HR segmentation masks are generated, at least one of which is optimized for registration to the LR MRI scan, and at least one of which if optimized for the training of the NN.

[0017] In some examples, the generating of the HR output from the input LR MRI scan using the trained NN includes increasing the resolution of the input LR MRI scan. In some examples, the training of the NN includes training the NN on ground truth segmentation masks generated from training input LR MRI scans whose resolution has been increased.

[0018] In some examples, the LR MRI input scan includes a spinal vertebra.

[0019] In some examples, a method further comprises loading the HR output intraoperatively and registering the HR output to an intraoperative image. In some examples, the HR output represents spinal vertebra, and the registering is performed separately for each vertebra.

[0020] In some examples, the input to the NN includes an interpolation confidence map. In some examples, the input to the NN includes a spinal vertebra and a representation of the level of such spinal vertebra.

[0021] In some examples, the training of the NN includes using a digitally reconstructed radiograph (DRR)-based loss function and / or a simulated registration-based loss function. In some examples, the simulated registration-based loss function includes an element segmentation-based term.

[0022] In some examples, the training of the NN includes generating a variety of simulated LR MRI scans from HR MRI scans and using the variety of simulated LR MRI scans as inputs to the NN during the training.

[0023] In some examples, the generating of the HR output includes segmenting an entire spine, dividing the spine into vertebral levels, and generating the HR output separately for each vertebral level.

[0024] In some examples, the input LR MRI scan comprises an MRI scan with thick slices, the HR output comprises a plurality of thin slices corresponding to each thick MRI scan slice, and the training a NN includes training the NN to interpolate values in the thin slices.

[0025] In some examples, the HR output includes areas which are not in a field of view (FOV) of the input LR MRI scan. In some examples, a method further comprises generating a statistical model of vertebral anatomy using a database of 3D vertebra representations which include areas that are typically not in the FOV of the input LR MRI scan, and applying such statistical model to the HR output to create areas of the HR output that are not in the FOV of the input LR MRI scan. In some examples, the input LR MRI scan is of the spine, and the areas that are typically not in the FOV of the input LR MRI scan include vertebral processes or portions thereof.

[0026] In some examples, the input to the NN further includes patient metadata.

[0027] In some examples, the input LR MRI scan comprises two or more views and / or sequences and / or protocols. In some examples, the two or more views and / or sequences and / or protocols are used as multiple input channels to the NN. In some examples, the two or more views and / or sequences and / or protocols are used separately as inputs to two or more NNs, and the generating of the HR output includes merging the outputs of the two or more NNs. In some examples, the two or more views and / or sequences and / or protocols are co-registered using a co-registration algorithm.

[0028] In some examples, the training of the NN includes training the NN to co-register the two or more views and / or sequences and / or protocols. In some examples, the two or more views and / or sequences and / or protocols are used to manually create a HR segmentation mask for the training of the NN.

[0029] In some examples, the HR segmentation mask represents a bone or an element thereof. In some examples, the HR segmentation mask includes labels for at least one bone element and one soft tissue element. In some examples, the soft tissue element includes one of a nerve, a spinal disc, a ligament, a muscle, or a tumour.

[0030] In some examples, the HR output includes representation of bone elements from a CT scan and representation of soft tissue elements from the input LR MRI scan.

[0031] In some exemplary embodiments, there is provided an apparatus, comprising: a MRI imaging system for providing MRI scans; and a computer system that includes a processor configured to train a NN with inputs that include training LR MRI scans provided by the MRI imaging system, and to generate a HR output directly from an input LR MRI scan selected from the group consisting of a training input LR MRI scan and a new input LR MRI scan using the trained NN, wherein a spacing of the HR output is smaller by at least 1mm along at least one axis compared to a spacing in the input LR MRI scan along the respective at least one axis.

[0032] In some examples, an apparatus further comprises a CT imaging system for providing CT scans to the computer system.

[0033] BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Non-limiting examples of embodiments disclosed herein are described below with reference to figures attached hereto that are listed following this paragraph. The drawings and descriptions are meant to illuminate and clarify embodiments disclosed herein and should not be considered limiting in any way. Like elements in different drawings may be indicated by like numerals. Elements in the drawings are not necessarily drawn to scale. In the drawings:

[0035] FIG. 1 shows a known art method for generating a HR segmentation mask from a HR CT scan; FIG. 2 shows one embodiment of a method for generating a HR output from a low- resolution MRI scan according to some aspects of the disclosure;

[0036] FIG. 3 shows an embodiment of a method for applying a trained NN as in step 204 of FIG 2, according to some aspects of the disclosure;

[0037] FIG. 4 shows an embodiment of a method for applying a trained NN as in step 204 of FIG 2, according to some aspects of the disclosure;

[0038] FIG. 5 shows another embodiment of a method for applying a trained NN as in step 204 of FIG 2, according to some aspects of the disclosure;

[0039] FIG. 6 shows one embodiment of a method for registering a HR segmentation mask as in step 508 of FIG 5, according to some aspects of the disclosure;

[0040] FIG. 7 shows another embodiment of a method for applying a trained NN as in step 204 of FIG 2, according to some aspects of the disclosure;

[0041] FIG. 8 shows one embodiment of a method for training a NN using LR MRI scans and unpaired HR scans as in step 706 of FIG 7, according to some aspects of the disclosure;

[0042] FIG. 9 shows another embodiment of a method for training a NN using LR MRI scans and unpaired HR scans as in step 706 of FIG 7, according to some aspects of the disclosure;

[0043] FIG. 10 shows another embodiment of a method for training a NN using LR MRI scans and unpaired HR scans as in step 706 of FIG 7, according to some aspects of the disclosure;

[0044] FIG. 11 shows another embodiment of a method for training a NN using LR MRI scans and unpaired HR scans as in step 706 of FIG 7, according to some aspects of the disclosure;

[0045] FIG. 12 shows another embodiment of a method for applying a trained NN as in step 204 of FIG 2, according to some aspects of the disclosure;

[0046] FIG. 13 shows another embodiment of a method for applying a trained NN as in step 204 of FIG 2, according to some aspects of the disclosure;

[0047] FIG. 14 shows an example of a NN design for use in the method described in FIG. 9;

[0048] FIG. 15 shows an example of a NN design for use in the method described in FIG. 10;

[0049] FIG. 16 shows an example of a NN design for use in the method described in FIG. 11;

[0050] FIG. 17 shows an example spinal LR MRI scan with the resulting HR segmentation mask and 3D bone model as per the method described in FIG. 2;

[0051] FIG. 18 shows an example output of the method described in FIG. 2 in the form of a HR artificial CT-like scan;

[0052] FIG. 19 shows a cross section of an exemplary LR MRI scan and a corresponding paired HR MRI scan; FIG. 20 shows cross sections of an exemplary LR MRI scan and a corresponding paired low resolution MRI scan of a different view;

[0053] FIG. 21 shows a cross section of an example sagittal LR MRI scan of a single vertebrae, and segmentation masks before and after post processing;

[0054] FIG. 22 shows a diagram for creating an up-sampling confidence map used as input to a trained NN according to some aspects of the disclosure;

[0055] FIG. 23 shows an example of a pedicle screw placement procedure in spinal surgery;

[0056] FIG. 24 shows one embodiment of a system for generating a HR output from a low- resolution MRI scan according to some aspects of the disclosure.

[0057] DETAILED DESCRIPTION

[0058] FIG. 2 shows one embodiment of a method for generating a HR output from a LR MRI scan according to some aspects of the disclosure. In step 202 a LR MRI scan is provided, and in step 204 a trained NN is applied to the LR MRI to generate a HR output. In what follows, the term “trained NN” is meant to apply to any computer algorithm where at least one part of the processing pipeline is performed by a trained NN, but which may also include additional non-deep learning algorithms (also known as “classical” algorithms) to perform pre-processing of the input to the trained NN, and / or post processing of the output of the trained NN.

[0059] The HR output of step 204 should be of sufficiently high resolution and provide sufficient detail of the bone structure relevant to a specific orthopaedic procedure, such that the HR output can be directly used in pre-op planning, and / or for registration to a scan obtained intraoperatively, and / or for intraoperative navigation. An example of such an output is shown in FIG. 17 (1704) and FIG. 18. Optionally, the registration to an intraoperative scan may be performed separately for each bone element, in a similar manner to that shown in FIG. 6 below. Optionally, a new HR output may be created from the outputs that were separately registered for each bone element.

[0060] Optionally, the HR output may include one or more of a HR segmentation mask, a HR artificial CT-like image (i.e. an output that resembles a CT scan for all bone elements relevant to the procedure being targeted, and which can be used by a clinical expert and / or computer algorithm instead of a real CT scan) such as that shown in FIG. 18, or a HR 3D bone model such as that shown in 1706 in FIG. 17. Optionally, a HR segmentation mask may include multiple bone elements, part of which may have the same segmentation label. For example, the HR segmentation mask may include multiple spine vertebrae all having the same label. Optionally the HR segmentation mask may include multiple bone elements with different labels. For example, it may include multiple spine vertebrae, each marked with a different label, as shown in 1704 of FIG. 17, so that the level of the vertebra can be identified for further processing. Optionally, different labels may be applied to different sub-parts of the same bone element. For example, if the bone element is a spinal vertebra, different labels may be used to mark the various parts of the vertebra, such as the pedicle and spinal processes.

[0061] Optionally, the HR output may be further processed to create another HR output. Optionally, such further processing may be performed using a classical algorithm. Optionally the further processing may be performed using an additional NN.

[0062] Optionally, the HR output may be a HR segmentation mask that can be further processed to create a HR artificial CT- like image such as that shown in FIG 18 and / or a HR 3D bone model such as that shown in 1706 of FIG. 17. Optionally, the artificial CT- like image may be created using an additional NN, resulting in an image which contains more detail and more closely resembles a real HR CT scan compared to an artificial CT-like scan created using a classical algorithm. Optionally, the additional NN may be trained using real CT scans as ground truth (GT) and HR segmentation masks generated from these CT scans as input. Optionally adversarial training may be utilized to improve the resulting HR artificial-like CT images.

[0063] Optionally, the trained NN in step 204 may receive at least one additional input in addition to the LR MRI scan provided in step 202. Optionally, the additional input may include metadata related to the patient. The metadata may include, but is not limited to, at least one of the patient's age, gender, physical characteristics such as height and weight, or medical history. Optionally, the additional input may include information about the relevant bone structure being segmented. For example, if the purpose of the trained NN is to segment a spinal vertebra, the additional input may include the level of the vertebra.

[0064] Optionally, the additional input may include another paired LR scan. Hereinafter, the term “paired means that the scan includes at least part of the same bone structure of the same patient as the LR MRI scan. For example, the additional input may include an additional LR MRI scan performed using a different protocol and / or of a different view to that provided in step 202. For example, one of the LR MRI scans may be a T1 scan, while another of the LR MRI scans may be a T2 scan. For example, one of the LR MRI scans may be of an axial view while another may be of a sagittal view. Optionally, the additional input may include a confidence map as described in FIG. 22 below, encoding the distance and / or angle between each grid point of the HR output and the nearest grid point of the input LR MRI scan. Optionally, the additional input may be provided as at least one additional input channel to the trained NN with the same dimensions as the LR MRI scan provided in step 202. Optionally, the additional input channel may encode numerical non-image data such as patient age, or spine vertebra level. Optionally, this numerical data may be encoded as a constant tensor. Optionally, this numerical data may be encoded using a NN embedding layer known in the art, whose weights can be trained.

[0065] FIG. 3 shows an embodiment of a method for applying a trained NN as in step 204. In step 302, a training dataset including LR MRI scans is provided. Optionally, the training dataset may include patient metadata as described above. Optionally the training dataset may include additional information about the bone structure of the LR MRI scan. Optionally, the training dataset may include additional paired LR and / or paired HR scans (for example HR CT scans and / or HR MRI scans) used as additional inputs to the trained NN in step 204, and / or used to generate ground truth (GT) for training the NN.

[0066] Optionally, the training dataset may include unpaired scans which are used to train the NN. Hereinafter, the term “unpaired scans” means that the scan includes at least part of the bone structure that the LR MRI scan includes, but of a different patient. Optionally, the training dataset may include data set aside for validating and / or testing the NN, as known to the art.

[0067] In step 304, a GT HR output is generated for, and registered to each LR MRI scan provided in step 302. Optionally, the GT HR output may be a HR segmentation mask manually segmented from an up-sampled version of the LR MRI scan. Optionally, the manual segmentation may be performed with the aid of an additional paired LR or HR scan. Optionally, the additional paired scan may be one of a HR CT scan, a HR MRI scan, a LR MRI scan of a different protocol, or a LR MRI scan of a different view. Optionally, the GT HR output may be another paired HR scan. Optionally, the GT HR output may be a HR CT scan and the desired output of step 204 may be a HR artificial CT- like image. Optionally, the GT HR output may be a HR segmentation mask which is automatically segmented from a paired HR scan, and the HR segmentation mask may be then registered to the LR MRI scan. Optionally, the automatic segmentation may be performed by a trained segmentation NN. Optionally, the registration may be performed by first segmenting the LR MRI, and then registering the two segmentation masks. Optionally, the registration may be performed by a trained NN. Optionally, the registration may be performed for each different element of the bone structure, as described in FIG. 6 below, and then combined to create the GT HR output. In step 306, the NN is trained using the LR MRI scan provided in step 302 and the GT HR output provided in step 304. Finally, in step 308 the trained NN is applied to a new LR MRI scan (not included in the training dataset provided in step 302) to generate the HR output.

[0068] Optionally, the input LR MRI scan to the NN in steps 306 and 308 may be pre-processed before input. Optionally, this pre-processing may include one of up-sampling, transforming to a fixed orientation, normalizing voxel values, clipping voxel values, changing the brightness, or changing the contrast. Optionally, up- sampling may be performed by another trained NN. Optionally, the output of the NN in steps 306 and 308 may be post-processed. Optionally, the post-processing may include, but is not limited to, at least one of smoothing, filling holes in a segmentation mask, removing disconnected components of a segmentation mask, extrapolating regions of a segmentation mask that are not in the field of view of the input LR MRI scan, or adding additional elements to a segmentation mask. Optionally, the post-processing may include classical non-deep learning algorithms. Optionally, the post-processing may include applying an additional trained NN.

[0069] FIG. 4 shows another embodiment of a method for applying a trained NN as in step 204. In step 402, a training dataset is provided including LR MRI scans, similar to step 302 in Fig 3. In step 404 the LR MRI scans are up-sampled. Optionally the up-sampling may be performed using an interpolation algorithm known in the art. Optionally, the up-sampling may be performed in stages. For example, given a LR MRI scan with spacing of 1x1x4mm, the initial stage may up-sample the scan to 1x1x2 mm, while a second stage will up-sample it to Ixlxlmm. Optionally, the up-sampling, or one stage of the up-sampling, may be performed by another trained NN. Optionally the up-sampling NN may be trained using paired HR MRI scans as GT. Optionally, the up-sampling NN may be trained using unpaired HR MRI scans and down- sampled versions of these unpaired scans.

[0070] Optionally, the input to the up-sampling NN may include a confidence map as described in FIG. 22 below, which encodes the distance from the final up-sampled grid point to the nearest grid point of the original LR MRI scan. Optionally, the input to the up-sampling NN may include information regarding spacing and or slice thickness of the input LR MRI scan.

[0071] In step 406, GT HR segmentation masks are manually segmented from the up- sampled LR MRI scan. Optionally, the manual segmentation may be performed with reference to another paired scan that provides information regarding the bone structure that is not shown, or not shown in sufficient detail and / or clarity, in the up- sampled LR MRI scan. Optionally, the manual segmentation may be performed by a trained clinical expert who is able to fill in parts of the bone structure not clearly visible in the up-sampled LR MRI scan. In step 408, similar to step 306, the NN is trained using LR MRI scans and their respective GT HR segmentation masks. Optionally, the LR MRI scans may be up- sampled before input to the NN as described in step 404. Optionally, the up-sampling process for the purpose of input to the NN in step 408 may be different to the up-sampling process for the purpose of manual segmentation in step 406. Optionally, a loss function comprising at least one of cross entropy, dice loss, or derivatives thereof may be used to train the NN. Optionally, more weight may be given in the loss function to regions on the boundaries of one or more of the segments in the segmentation mask. Optionally, more weight may be given in the loss functions to regions or points of the segmentation mask that have more clinical significance in the relevant procedure. For example, FIG. 23 below illustrates screw placement in a spinal vertebra where the width of the pedicle is of critical significance. Thus, extra weight may be provided in the loss function to the pedicle and / or the area surrounding it. Optionally, the loss function may include a differentiable process to create an artificial CT-like image from the output segmentation mask and then a comparison of this image to a paired HR CT scan Such a loss function optimizes the use of the NN output during intraoperative registration to a fluoro image generated for example by C-arm imaging.

[0072] Optionally, a digitally reconstructed radiograph (DRR) may be derived using a differentiable process from a paired HR CT scan and compared to cross sections of the artificial CT-like image. Optionally, an affine perturbation may be applied to the artificial CT-like image followed by a differentiable registration process to re-register the perturbed artificial CT-like image to the paired HR CT scan, followed by a comparison of the registration output to the original orientation before perturbation. This latter option further optimizes the use of the NN output during intraoperative registration. Finally, in step 410, similar to step 308, the trained NN is applied to a new LR MRI scan to generate a HR segmentation mask.

[0073] FIG. 5 shows yet another embodiment of a method for applying a trained NN as in step 204. In step 502, a training dataset is provided including LR MRI scans and paired HR scans. In all other aspects, step 502 is similar to step 302. Optionally, the HR scans may be CT scans. Optionally, the HR scans may be MRI scans. In step 504, similar to step 404, the LR MRI scans are up-sampled. In step 506, GT HR segmentation masks are created from the paired HR scans. Optionally, the GT HR segmentation mask may include soft tissue elements taken from the up- sampled LR MRI scan, and the NN in step 204 is trained to output these soft tissue elements as part of the HR output. Optionally the LR MRI scan may be a spinal scan and the soft tissue elements include one of the spinal cord, nerves, spinal discs, muscle, ligaments, or tumours. Optionally the GT HR segmentation mask may be manually segmented from the paired HR scan. Optionally, the HR segmentation mask may be automatically segmented from the paired HR scan. Optionally, the automatic segmentation may be performed by a trained segmentation NN.

[0074] In step 508, the GT HR segmentation mask from step 506 is registered to the up-sampled LR MRI scan from step 504. Optionally, the registration may be performed by a classical algorithm. Optionally the registration may be performed by a NN. Optionally, the up- sampled LR MRI scan may be first segmented to create another segmentation mask, which is then registered to the GT HR segmentation mask.

[0075] Optionally, an iterative process may be performed whereby a LR MRI scan segmentation mask may be generated manually or using a pre-existing algorithm in the first iteration, and then in subsequent iterations the output of the previous iteration of step 204 may be used to create the LR MRI scan segmentation mask, thus improving the registration process for the current iteration. Optionally, the optimization function of the registration process may include contributions from segmentation masks and / or grey scale-based comparison of the original images used to create the segmentation mask. Optionally, more weight may be provided in the registration optimization function to regions of the segmentation mask which have more clinical significance to the relevant procedure, as for example illustrated by FIG. 23 below. Optionally, the optimization function of the registration process may use multiple labels from the segmentation masks rather than a single label. Optionally, the registration may be performed for each different element of the bone structure, as described in FIG. 6 below, and then combined to create the GT HR output.

[0076] Optionally, step 506 may generate two or more types of HR segmentation masks, at least one of which is optimized for the purpose of registration in step 508 and at least one of which is optimized for training the NN in step 510 below.

[0077] In step 510, similar to step 306, the NN is trained using LR MRI scans and their respective GT HR segmentation masks. In step 512, similar to step 308, the trained NN is applied to a new LR MRI scan to generate a HR segmentation mask.

[0078] FIG. 6 shows one embodiment of a method for performing the registration in step 508. Given that the LR MRI scan and the paired HR CT scan used to generate the GT HR segmentation mask were typically acquired at different time points and possibly with different patient postures, the different elements of the relevant bone structure may have different positions and / or orientations one to another in each of the two scans. For example, in spinal scans the relation between the various vertebrae may be different in the HR CT scan and the LR MRI scan. In such cases, an element-by-element registration as described below is preferred.

[0079] In step 602, the up- sampled LR MRI scan is first segmented to create a segmentation mask, similar to the process described in relation to FIG. 5 above. In step 604, the segmentation mask from step 602 is separated into elements. For example, a segmentation mask of a spine may be separated into different vertebrae. Optionally, the segmentation mask created in step 602 may have different segmentation labels which facilitate the separation into elements. Optionally, separation into elements may be performed by another NN. In step 606, the GT HR segmentation mask from step 506 is separated into elements, in a similar manner to step 604. In step 608, each element from step 604 is separately registered to the corresponding element from step 606. This registration can be carried out using standard methods known to the art since each separate bone segment is rigid. When necessary, additional methods can be used as described in relation to FIG. 5 above. In step 610, the separate registered bone elements from step 608 are combined to create a single registered GT HR segmentation mask, which is the output of step 508 above.

[0080] FIG. 7 shows yet another embodiment of a method for applying a trained NN as in step 204. This method is similar to that described in FIG. 5 with the main difference being that the training data includes unpaired HR scans. This is particularly relevant when insufficient LR MRI scans and paired HR scans are available, or when it is too costly or time consuming to generate sufficient numbers of LR MRI scans and paired HR scans. Step 702 is similar to step 502, with the addition that the training data now includes unpaired HR scans. Optionally, the unpaired HR scans may be MRI scans. Optionally, the unpaired HR scans may be CT scans. Step 704 is similar to step 508, with the difference that it is only applied to the LR MRI scans and to the corresponding paired HR scans, but not to the unpaired HR scans. Step 706 is similar to step 510, with the addition that the unpaired HR scans are also used in the training process. Step 708 is similar to step 512.

[0081] FIG. 8 shows an embodiment of a method for training a NN using unpaired HR scans as in step 706. Optional step 802 may rescale each LR MRI scan and / or each HR scan so that all the scans have the same resolution. Optionally, the LR scan may be up-sampled similarly to step 404. Optional step 804 may process the HR scan to look like a MRI scan. Optionally, step 804 may be performed using a generative adversarial network (GAN) as known in the art. In step 806, the NN is pre-trained using the unpaired HR scans and their corresponding GT HR segmentation masks, and then in step 808 the NN is further trained in a similar manner to step 510 using the LR MRI scans and their corresponding GT HR segmentation masks. Optionally, only part of the NN weights may be pre-trained using the unpaired HR scans in step 806, while the remaining NN weights may be randomly initialized in step 808. Pre-training step 806 allows the NN to learn details of the relevant bone structure before training with LR MRI scans in step 808, and thus to achieve overall better results.

[0082] FIG. 9 shows another embodiment of a method for training a NN using unpaired HR scans as in step 706. Step 902 uses alternately batches of LR MRI scans with their corresponding GT HR segmentation masks, and batches of HR scans with their corresponding GT HR segmentation masks, to train a NN which has some weights and / or nodes relevant only to the HR scan input, other weights and / or nodes only relevant to the LR MRI scan input, and still other weights and / or nodes which are common to both types of input. FIG. 14 below shows an example of such a NN. By alternatively using batches of HR scans and batches of LR MRI scans, the common weights will learn only the common features and structures, while ignoring those that are specific to each imaging modality. Optionally, the majority of the weights and nodes may be common, while only weights and nodes used for batch and / or instance normalization are separate for the two types of input in one or more of the NN layers. In step 904, the trained NN is generated by keeping only weights and nodes that are relevant to LR MRI scans.

[0083] FIG. 10 shows yet another embodiment of a method for training a NN using unpaired HR scans as in step 706. In step 1002, a first NN is trained in a similar manner to step 510. In step 1004 a second NN Is trained which takes as input a HR segmentation mask and outputs the same HR segmentation mask. Optionally, the second NN may be an autoencoder as for example shown in FIG. 15. Optionally, the input HR segmentation mask may be processed to include noise and / or non-smooth elements and / or holes and / or missing parts, while the NN is trained to reproduce the original segmentation mask. Optionally, the second network may be trained by generating GT HR segmentation masks similar to step 506. In step 1006, the trained NN is created by concatenating the second NN to the first NN, such that the output of the first NN is used as the input to the second NN. Thus, the second NN is used to improve the segmentation mask generated by the first NN. In an optional step 1008, the trained NN may be further fine tuned by training it end-to-end with LR MRI scans and paired GT HR segmentation masks.

[0084] FIG. 11 shows another embodiment of a method for training a NN using unpaired HR scans as in step 706. In step 1102, a first NN is trained in a similar manner to step 510. In step 1104, a second NN is trained. The second NN accepts as input a HR segmentation mask and determines whether it was generated using the first NN or generated directly from a HR scan. Using the nomenclature of adversarial training, the first NN corresponds to a generator (and marked accordingly in step 1102), while the second NN corresponds to a discriminator (marked accordingly in step 1106). In step 1104, training of the discriminator consists of alternately providing it with HR segmentation masks generated from unpaired HR scans, together with a GT “Valid” label, and with outputs of the generator trained in step 1102 together with a GT “Invalid” label. Thus, the discriminator learns to identify between the two types of segmentation masks. In step 1106, following the initial training of the generator in step 1102 and of the discriminator in step 1104, the two NNs are trained in an adversarial manner in step 1106 as known in the art, so that the generator learns to generate better HR segmentation masks, while the discriminator learns to better identify between HR segmentation masks originating in a HR scan, and those output by the generator. In step 1108, the trained NN is created by retaining only the generator and discarding the discriminator.

[0085] FIG. 12 shows another embodiment of a method for applying a trained NN as in step 204. Only steps 1202 and 1204 are shown, while steps 306 and 308 (not shown) follow after step 1204. In step 1202, a training dataset is provided including first LR MRI scans and paired second LR MRI scans, where each scan is performed along a different axis. In all other aspects, step 1202 is similar to step 302. Optionally, the LR MRI scans may be spinal scans, or one of the LR MRI scans may be an axial scan while the other may be a sagittal scan, as for example illustrated in FIG. 20 below. In step 1204, a GT HR output is generated and registered to each respective first LR MRI scan using the second LR MRI scan. Optionally, this may be achieved by manually segmenting the first MRI scan, up-sampling the resulting segmentation mask, and manually interpolating the segmentation between the slices along the low-resolution axis of the first MRI scan, using the second MRI scan as a reference. Optionally, this may be achieved by co-registering the two MRI scans, creating a combined HR MRI scan, and segmenting the resulting HR MRI scan manually or automatically. Optionally, this may be achieved by creating a LR segmentation mask from each of the two MRI scans, co-registering the LR segmentation masks, and creating a combined HR segmentation mask. Optionally, the task of separately segmenting each LR MRI scan may be performed by a trained NN as in step 204 which is optimized for the corresponding scan direction, and the improved GT HR segmentation mask generated in step 1204 may be used to improve the performance of the trained NN designed for the view of the first LR MRI scan. By reversing the roles of the first and second LR MRI scans, the trained NN designed for the view of the second LR MRI scan may also be improved. This process may be performed iteratively, where in each iteration the trained NNs from the previous iterations are used to generate segmentation masks prior to step 1204, and the resulting improved GT HR segmentation masks generated by step 1204 are used to improve the performance of the trained NNs in the current iteration.

[0086] FIG. 13 shows another embodiment of a method for applying a trained NN as in step 204. Step 1302 is similar to step 302. Step 1304 is similar to step 304 with the addition that the GT HR segmentation masks also include at least one label for a bone cortex. Step 1306 is similar to step 306. However, since the bone cortex is more difficult to identify on a MRI scan than on other parts of the bone, it is expected that the output HR segmentation mask of the NN trained in step 1306 will not include the bone cortex or will include it only partially. To address this, in step 1308 a second NN is trained. The second NN takes as input the LR MRI scan and the segmentation mask produced in step 1306. The information included in the input segmentation mask guides the second NN regarding the location of the other bone structures, and thus allows it to infer the location of the bone cortex and correctly segment it. Thus, the output of step 1308 is a more complete segmentation mask including the bone cortex. Optionally, patient metadata may also be used as input to the NN in step 1308 to further facilitate the bone cortex segmentation. Such metadata may include, but is not limited to, the patient's age, gender, physical characteristics such as height and weight, and medical history. Step 1310 creates a trained (combined) NN by concatenating the first NN from step 1306 with the second NN from step 1308. Step 1312 is similar to step 308, with the difference that the output segmentation mask includes the bone cortex.

[0087] FIG. 14 shows an example of a NN design that may be used in step 902, This NN comprises two paths, a “CT path” and a “MRI path”. The CT path begins with an input layer 1402, which accepts as input an unpaired HR CT scan. This is followed by a “CT path” encoder section 1404, followed by a common section 1406, followed by a “CT path” decoder section 1408 and ending in a “CT path” output layer 1410. The GT used to train the “CT path” can be generated using for example step 506. The encoder and decoder layers may be based on any of the variations known in the art. The common section may comprise one or more deep layers. In some embodiments, encoder section 1404 may also be connected directly to decoder section 1408 in addition to being connected to common section 1406.

[0088] The “MRI path” includes sections 1412, 1414, 1418 and 1420, which correspond to CT sections 1402, 1404, 1408 and 1410 respectively. These sections are typically similar to their corresponding CT sections but may have differences designed to learn the relevant features of each modality. The input to the “MRI path” is the LR MRI scan or an up-sampled LR MRI scan, and the GT used to train the “MRI path” can be generated using for example step 304 FIG. 15 shows an example of a NN design that can be used in the method described in FIG. 10. An input LR MRI scan 1502 is fed to a segmentation NN 1504 which outputs an intermediate segmentation mask 1506. Segmentation NN 1504 may, for example, be similar to the trained NN used in step 204. Intermediate segmentation mask 1506 is provided as input to an autoencoder 1508 which outputs a final segmentation mask 1510. Autoencoder 1508 comprises an encoder 1512 followed by a low dimensional feature layer 1514, followed by a decoder 1516. The training process of autoencoder 1508 involves providing it with valid segmentation masks as input, as for example manually generated from HR scans, and training it to reproduce the same segmentation mask as output. Through this training process, encoder 1512 learns to generate an abstract representation of a valid segmentation mask within low dimensional feature layer 1514, while decoder 1516 learns to reproduce the valid segmentation mask from this abstract representation. Then, given an imperfect or noisy segmentation mask at the input, the autoencoder will output a valid segmentation mask whose abstract representation is closest to the input. Optionally, to make the training process of autoencoder 1508 more robust, all or part of the input segmentation masks provided at the input may be corrupted, and the autoencoder may be trained to reproduce the original uncorrupted segmentation mask at the output. Preferably, the corrupted segmentation mask should emulate as best as possible typical imperfections generated by segmentation model 1504. Optionally, a final fine tuning training step similar to step 1008 may be performed on the combined NN comprising segmentation model 1504 and autoencoder 1508, using LR MRI scans as input and GT HR segmentation masks generated using for example step 304.

[0089] FIG. 16 shows a block diagram providing an example implementation of the method described in FIG. 11. Segmentation masks 1602 generated from HR scans are fed to a discriminator NN (or simply “discriminator”) 1608 together with a GT “Valid” label. Alternatively, LR MRI scans 1604 are fed to a segmentation mask generator NN (or simply “generator”) 1606, and the resulting segmentation mask is fed to discriminator 1608 together with a GT “Invalid” label. Generator 1606 may, for example, be similar to the trained NN used in step 204. When discriminator 1608 correctly identifies the label corresponding to the input segmentation mask, positive training feedback is provided to discriminator 1608, while negative training feedback is provided to generator 1606. When discriminator 1608 incorrectly identifies the label corresponding to the input segmentation mask, negative training feedback is provided to discriminator 1608, while positive training feedback is provided to generator 1606. In this manner the discriminator becomes better at identifying the source of the input segmentation mask, while the generator becomes better at generating higher quality segmentation masks.

[0090] FIG. 17 shows a LR spinal MRI scan 1710, which is an example of a LR MRI scan provided in step 202, and a segmentation mask 1712 of the vertebrae, which is an example of the output generated by step 204. 3D model 1714 of the vertebrae may be generated from segmentation mask 1712 using methods known in the art, such as the marching cubes algorithm. In this example, segmentation mask 1712 has different labels (represented by different shades of grey) for each shown vertebra, thus identifying the position of the vertebra (in this example, LI, L2, L3, L4, L5 and SI). Optionally, the output of step 204 may provide a segmentation mask where all vertebrae have the same label. Optionally, segmentation mask 1712 may be an intermediate output of step 204, which is then separated into individual vertebra. The separated vertebrae segments may then be used to crop the corresponding sections from MRI scan 1710 and provide them as separate input to a second part of the trained NN in step 204 which outputs separate refined segmentation masks for each vertebra. Optionally, before input to the second part of the NN, all vertebrae may be transformed to have the same orientation. Optionally, an input encoding the level of the vertebra may also be provided to the second part of the trained NN. Optionally, each such refined individual vertebra segmentation mask may then be used as input to a third part of the trained NN in step 204 which outputs a segmentation mask where different component elements of the vertebra have different segmentation labels. Optionally, these component elements may include, but are not limited to, the body, the cortex, the pedicles, the spinous process, the transverse process, or the lamina. Optionally, additional segmentation labels that represent soft tissue and not bone may be output. Optionally, soft tissue labels may include one of the spinal cord, nerves, spinal discs, muscle, ligaments or tumours. Optionally, the additional labels may be used to give more weight to certain regions of the segmentation mask in the loss function in the first and / or second part of the trained NN, as for example discussed in the description of FIG 4 above. Optionally, the additional labels may be used to give more weight to certain regions of the segmentation mask in the registration optimization function, as discussed in the description of FIG. 5 above. For example, the pedicle label may be used to apply additional weight to the loss function and / or the registration optimization function in the vicinity of the pedicle for screw placement procedures as for example shown in FIG. 23.

[0091] Optionally, an iterative process may be applied whereby the third part of the trained NN from a previous iteration may be used to fine tune the loss function of the first and / or second part of the trained NN in the current iteration, and / or the registration optimization function preceding the first part of the trained NN (e.g. step 508), thus improving the overall results of the entire trained NN. Optionally, the segmentation masks for the individual vertebra may be combined to create a final segmentation mask which includes refined segmentation masks for each vertebra optionally separated into component elements.

[0092] FIG. 18 shows different cross-sections of an example of a HR artificial CT-like image representing a spine. Optionally, such a CT-like image may be created from a segmentation mask such as 1710. Optionally, such a CT-like image may be a direct output of step 204. Optionally, artificial CT-like images may be used to register intraoperative C-arm images. Optionally, artificial CT-like images may be used by physicians who previously used real CT images for pre-op planning, instead of, or in addition to, HR segmentation masks.

[0093] FIG. 19 shows an example cross section of a LR MRI scan 1902 for a single spine vertebra, and the same cross section of a paired HR MRI scan 1904. LR MRI scan 1902 and paired HR MRI scan 1904 may be examples of the scans provided in step 502. These scans may also be examples of input and GT scans used to train a NN to up-sample LR MRI scans, as for example described in relation to step 504.

[0094] FIG. 20 shows axial cross section 2002 and sagittal cross section 2004 of an axial spinal LR MRI scan, and a corresponding axial cross section 2006 and sagittal cross section 2008 of a paired sagittal LR MRI scan. Such paired LR MRI scans of different views may be used for example as described in FIG 12 above.

[0095] FIG. 21 shows an example of an axial cross section of a sagittal LR MRI scan showing a single vertebra, with segmentation mask 2102 representing an example of an output of a trained NN before being post processing, and segmentation mask 2104 representing an example of an output of a trained NN after post processing. In this example, the post processing involves extrapolating the segmentation mask outside the field of view of the sagittal view LR MRI scan.

[0096] Optionally, extrapolating the segmentation mask outside the field of view of the LR MRI scan may involve creating a statistical model of the full segmentation mask, and then using known points within the field of view of the LR MRI scan to find the best fit of the statistical model, thus extrapolating the segmentation mask outside the field of view of the LR MRI scan. Optionally, the statistical model is created using a limited number of LR MRI scans paired with an additional scan that completes the field of view. Optionally, the additional scan may be a HR CT scan or HR MRI scan. Optionally, the additional scan may be another LR MRI scan. For example, axial cross section of axial scan 2002 shows the spinal processes which are not visible in the sagittal scan shown in FIG. 21. Thus, a limited number of paired axial LR MRI scans may be used to generate a statistical model of a HR segmentation mask which includes the spinal processes, with the statistical model being applied during post processing to produce a final segmentation mask including the spinal processes, even though they are not visible in a sagittal LR MRI scan used as input to step 204. Optionally, the statistical model may be used as input to an additional NN which outputs a final segmentation mask including the parts that were not visible in the input LR MRI scan. In this case, the additional NN refines the statistical model specifically for the input LR MRI scan.

[0097] FIG. 22 shows a diagram for creating a confidence map used as input to a trained NN according to some aspects of the disclosure. Referring to FIG. 2, the HR output of step 204 has higher resolution than the input LR MRI scan provided in step 202. In some cases, the HR output of step 204 may also have s different orientation compared to the input LR MRI scan. For example, if the input LR MRI scan is of a single spinal vertebra, then as part of step 204, it may be re-oriented so that the input to the trained NN always has a constant orientation. Thus, the HR output will have a different orientation compared to the input LR MRI scan of the vertebra. In FIG. 22, a grid 2202 is an example of a grid of the HR output, which is of higher resolution and at a different orientation compared to grid 2204. Optionally, the trained NN in step 204 may benefit from having as input a map that encodes the “confidence” of each point in output grid (for example grid 2202), whereby in this context confidence means how close it is to the nearest grid point in the input (for example a grid 2204). The assumption here is that the closer a point on the output grid is to a point on the input grid, the more accurate is the NN output for that point. Optionally, the trained NN in step 204 may benefit from having as input a map that encodes the direction (for example in the form of an angle or angles) of each point in the output grid to the nearest point in the input grid. Optionally, the confidence and / or direction of each point in the output grid may be used in the loss function when training the NN. For example, extra weight may be given in the loss function to points which have lower confidence.

[0098] To generate the confidence map and / or direction map, each point in the output grid (for example the grey point shown in grid 2202) is mapped to the corresponding point in the coordinate system of the input grid (in this example, the grey point in grid 2204). Then, the nearest point in the input grid is located, and the distance and / or direction to this nearest point is calculated. In this example, the direction and distance are encoded in the vector shown in grid 2204. Finally, the distance and / or direction are mapped back to the coordinate system of the output grid and used to encode the distance and / or direction of that particular grid point. In this example, the direction and distance are encoded in the vector shown in grid 2202, which corresponds to the vector shown in grid 2204 mapped back to the coordinate system of grid 2202.

[0099] FIG. 23 shows an example of a pedicle screw placement procedure in spinal surgery. Sub-figure 2302 shows a 3D model of the screw placement in the vertebra, while sub-figures 2304 and sub-figure 2306 show axial and sagittal cross sections of the screw placement respectively. In each of the sub-figures an arrow points to the pedicle, which is the narrowest section through which the screw passes. Therefore, it is critical that the HR output of 204 be as accurate as possible in the region of the pedicle for pre-op planning and / or intraoperative registration and / or intraoperative navigation. Preferably, all regions of the pedicle should have sub-mm accuracy. Optionally, more weight may be given to the pedicle relative to other parts of the vertebra in the loss function used during training of the NN in 204. Optionally, more weight may be given to the pedicle relative to other parts of the vertebra in the registration optimization function used for example in 508. Optionally, the physician planning the procedure may mark critical points and / or regions and / or dimensions on the vertebra which are critical to the procedure, and these markings are then used as input to the trained NN in step 204 in addition to the LR MRI scan. In this case, the same markings should be provided as input during training of the NN, thus allowing the NN to learn to provide higher accuracy in the corresponding regions of the vertebrae. Optionally, the markings can also be used in the loss function used to train the NN, so that the NN optimizes the accuracy of the prediction in the corresponding regions of the vertebra.

[0100] FIG. 24 shows one embodiment of a system for generating a HR output from a low- resolution MRI scan according to some aspects of the disclosure; MRI imaging sub-system 2402 is used to acquire MRI scans. Optionally these include LR MRI scans as in step 202 and step 302. Optionally these include HR MRI scans as in step 502 and in step 702. Optional CT imaging sub-system 2404 is used to acquire CT scans. Optionally, these include HR CT scans as in step 502 and in step 702. Processing unit 2406 is configured to apply a trained NN to a LR MRI scan to produce a HR output as in step 204. Computer screen 2408 is used to display the HR output generated by processing unit 2406. Optionally computer screen 2408 may used for the purpose of facilitating pre-op planning. Computer storage 2410 is used to store the HR output generated by processing unit 2406. Optionally, the stored HR output may be used at a later stage for pre-op planning and / or intraoperative registration and / or intraoperative navigation. Some stages (steps) of the aforementioned method(s) may also be implemented in a computer program for running on a computer system, at least including code portions for performing steps of the relevant method when run on a programmable apparatus, such as a computer system or enabling a programmable apparatus to perform functions of a device or system according to the disclosure. Such methods may also be implemented in a computer program for running on the computer system, at least including code portions that make a computer execute the steps of a method according to the disclosure.

[0101] A computer program is a list of instructions such as a particular application program and / or an operating system. The computer program may for instance include one or more of: a subroutine, a function, a procedure, a method, an implementation, an executable application, an applet, a servlet, a source code, code, a shared library / dynamic load library and / or other sequence of instructions designed for execution on a computer system.

[0102] The computer program may be stored internally on a non-transitory computer readable medium. All or some of the computer program may be provided on computer readable media permanently, removably or remotely coupled to an information processing system. The computer readable media may include, for example and without limitation, any number of the following: magnetic storage media including disk and tape storage media; optical storage media such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; non-volatile memory storage media including semiconductor-based memory units such as FLASH memory, EEPROM, EPROM, ROM; ferromagnetic digital memories; MRAM; volatile storage media including registers, buffers or caches, main memory, RAM, etc.

[0103] A computer process typically includes an executing (running) program or portion of a program, current program values and state information, and the resources used by the operating system to manage the execution of the process. An operating system (OS) is the software that manages the sharing of the resources of a computer and provides programmers with an interface used to access those resources. An operating system processes system data and user inputs and responds by allocating and managing tasks and internal system resources as a service to users and programs of the system.

[0104] The computer system may for instance include at least one processing unit, associated memory and a number of input / output (I / O) devices. When executing the computer program, the computer system processes information according to the computer program and produces resultant output information via I / O devices. Unless otherwise stated, the use of the expression “and / or” between the last two members of a list of options for selection indicates that a selection of one or more of the listed options is appropriate and may be made.

[0105] It should be understood that where the claims or specification refer to "a" or "an" element, such reference is not to be construed as there being only one of that element.

[0106] All references mentioned in this specification are herein incorporated in their entirety by reference into the specification, to the same extent as if each individual reference was specifically and individually indicated to be incorporated herein by reference. In addition, citation, or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present disclosure.

[0107] While this disclosure has been described in terms of certain embodiments and generally associated methods, alterations and permutations of the embodiments and methods will be apparent to those skilled in the art. The disclosure is to be understood as not limited by the specific embodiments described herein, but only by the scope of the appended claims.

Claims

WHAT IS CLAIMED IS:

1. An imaging method, comprising: training a neural network (NN) with inputs that include training low-resolution (LR) magnetic resonance imaging (MRI) scans; and generating a high-resolution (HR) output directly from an input LR MRI scan selected from the group consisting of a training input LR MRI scan and a new input LR MRI scan using the trained NN, wherein a spacing of the HR output is smaller by at least 1mm along at least one axis compared to a spacing in the input LR MRI scan along the respective at least one axis.

2. The method of claim 1, wherein the HR output includes an artificial computerized tomography (CT)-like scan.

3. The method of claim 2, wherein the artificial CT-like scan represents soft tissues with a shade or texture that is different to the bone representation.

4. The method of claim 1, wherein the HR output includes a segmentation mask.

5. The method of claim 4, further comprising generating a HR artificial computerized tomography (CT)-like scan from the HR segmentation mask.

6. The method of claim 4, wherein the training of the NN includes training the NN using CT data that includes computerized tomography (CT) scans and a HR segmentation mask created from the CT scans.

7. The method of claim 6, wherein the training of the NN includes at least one of: pretraining the NN using the CT scans, training a second NN using the CT data that includes shared weights with the NN, using an autoencoder trained from the CT data to post process the segmentation mask produced by the NN, or using the CT data to pretrain a discriminator, and training the discriminator together with the NN in an adversarial manner.

8. The method of claim 1, wherein the generating of the HR output directly from the input LR MRI scan using the trained NN includes acquiring a database of LR MRI scans andassociated CT scans belonging to the same patient, training the NN using the LR MRI scans as inputs and using the CT scans to generate ground truth for the HR output.

9. The method of claim 8, wherein the ground truth includes a HR segmentation mask registered to the training input LR MRI scan.

10. The method of claim 9, wherein the HR output is used to improve the registration, and wherein the process of registering the ground truth HR segmentation mask to the input LR MRI scan, training the NN using the registered scans, and using the HR output to improve the registration is repeated iteratively.

11. The method of claim 9, wherein the training input LR MRI scan and associated CT scans are spine scans, and wherein the registration is performed separately for each vertebra.

12. The method of claim 9, wherein two or more of the ground truth HR segmentation masks are generated, at least one of which is optimized for registration to the LR MRI scan, and at least one of which if optimized for the training of the NN.

13. The method of claim 1, wherein the generating of the HR output from the input LR MRI scan using the trained NN includes increasing the resolution of the input LR MRI scan.

14. The method of claim 13, wherein the training of the NN includes training the NN on ground truth segmentation masks generated from training input LR MRI scans whose resolution has been increased.

15. The method of claim 1, wherein the LR MRI input scan includes a spinal vertebra.

16. The method of claim 1, further comprising loading the HR output intraoperatively and registering the HR output to an intraoperative image.

17. The method of claim 16, wherein the HR output represents spinal vertebra, and wherein the registering is performed separately for each vertebra.

18. The method of claim 1, wherein the input to the NN includes an interpolation confidence map.

19. The method of claim 1, wherein the input to the NN includes a spinal vertebra and a representation of the level of such spinal vertebra.

20. The method of claim 1, wherein the training of the NN includes using a digitally reconstructed radiograph (DRR)-based loss function and / or a simulated registration-based loss function.

21. The method of claim 20, wherein the simulated registration-based loss function includes an element segmentation-based term.

22. The method of claim 1, wherein the training of the NN includes generating a variety of simulated LR MRI scans from HR MRI scans and using the variety of simulated LR MRI scans as inputs to the NN during the training.

23. The method of claim 1, wherein the generating of the HR output includes segmenting an entire spine, dividing the spine into vertebral levels, and generating the HR output separately for each vertebral level.

24. The method of claim 1, wherein the input LR MRI scan comprises an MRI scan with thick slices, wherein the HR output comprises a plurality of thin slices corresponding to each thick MRI scan slice, and wherein the training a NN includes training the NN to interpolate values in the thin slices.

25. The method of claim 1, wherein the HR output includes areas which are not in a field of view (FOV) of the input LR MRI scan.

26. The method of claim 25, further comprising generating a statistical model of vertebral anatomy using a database of 3D vertebra representations which include areas that are typically not in the FOV of the input LR MRI scan, and applying such statistical model to the HR output to create areas of the HR output that are not in the FOV of the input LR MRI scan.

27. The method of claim 26, wherein the input LR MRI scan is of the spine, and wherein the areas that are typically not in the FOV of the input LR MRI scan include vertebral processes or portions thereof.

28. The method of claim 1, wherein the input to the NN includes patient metadata.

29. The method of claim 1, wherein the input LR MRI scan comprises two or more views and / or sequences and / or protocols.

30. The method of claim 29, wherein the two or more views and / or sequences and / or protocols are used as multiple input channels to the NN.

31. The method of claim 29, wherein the two or more views and / or sequences and / or protocols are used separately as inputs to two or more NNs, and wherein the generating of the HR output includes merging the outputs of the two or more NNs.

32. The method of claim 29, wherein the two or more views and / or sequences and / or protocols are co-registered using a co-registration algorithm.

33. The method of claim 29, wherein the training of the NN includes training the NN to coregister the two or more views and / or sequences and / or protocols.

34. The method of claim 29, wherein the two or more views and / or sequences and / or protocols are used to manually create a HR segmentation mask for the training of the NN.

35. The method of claim 3, wherein the HR segmentation mask represents a bone or an element thereof.

36. The method of claim 3, wherein the HR segmentation mask includes labels for at least one bone element and one soft tissue element.

37. The method of claim 36, wherein the soft tissue element includes one of a nerve, a spinal disc, a ligament, a muscle, or a tumour.

38. The method of claim 36, wherein the HR output includes representation of bone elements from a CT scan and representation of soft tissue elements from the input LR MRI scan.

39. An apparatus, comprising: a magnetic resonance imaging (MRI) imaging system for providing MRI scans; and a computer system that includes a processor configured to train a neural network (NN) with inputs that include training low-resolution (LR) MRI scans provided by the MRI imaging system, and to generate a high-resolution (HR) output directly from an input LR MRI scan selected from the group consisting of a training input LR MRI scan and a new input LR MRI scan using the trained NN, wherein a spacing of the HR output is smaller by at least 1mm along at least one axis compared to a spacing in the input LR MRI scan along the respective at least one axis.

40. The apparatus of claim 39, further comprising a computerized tomography (CT) imaging system for providing CT scans to the computer system.

Images