Method for generating 3D printable models of patient-specific anatomy

Abstract

A computer-implemented method for generating a 3D printable model of a patient-specific anatomical feature from a 2D medical image is provided. A 3D image is automatically generated from a set of 2D medical images. The generated 3D image is segmented using a machine learning-based image segmentation technique. A 3D printable model of the patient-specific anatomical feature is created from the segmented 3D image.

Description

Technical Field

[0001] The field of the invention relates to a computer-implemented method for generating a 3D printable model of a patient-specific anatomy based on a 2D medical image.

[0002] Portions of the disclosure of this patent document contain material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. Background Art

[0003] Creating accurate 3D printed models of specific parts of a patient's anatomy helps transform surgical procedures by providing clinicians with insights for preoperative planning. Benefits include, for example, better clinical outcomes for patients, reduced surgical time and costs, and the ability for patients to better understand the planned surgery.

[0004] However, there is still a need to provide a secure platform that can order and deliver 3D printed models in a timely and customizable manner. In addition, there is a need to provide 3D printable models that provide better insights into patient anatomy or pathology. Summary of the invention

[0005] A computer-implemented method for generating a 3D printable model of a patient-specific anatomical feature from 2D medical images is provided, wherein a 3D image is automatically generated from a set of 2D medical images, the generated 3D image is segmented using a machine learning-based image segmentation technique, and the 3D printable model of the patient-specific anatomical feature is created from the segmented 3D image.

[0006] Optional features in embodiments of the present invention include any one or more of the following:

[0007] • The set of 2D medical images are images of the patient acquired from one or a combination of: CT, MRI, PET and / or SPCET scanners.

[0008] Simultaneously process 2D medical images from multiple scanning technologies.

[0009] - Automatically pre-processing the set of 2D medical images such that important or critical features of the patient-specific anatomical features are made visible in the 3D printable model.

[0010] • Pre-processing of the 2D medical images is based on the specific anatomical features, the patient's specific pathology or any downstream application, such as pre-operative planning or training purposes.

[0011] The set of 2D medical images is pre-processed to generate a set of new 2D medical images uniformly distributed according to a predetermined orientation.

[0012] • The predetermined orientation is based on the patient specific anatomical features, the patient specific pathology or any downstream application, such as pre-operative planning or training purposes.

[0013] • The predetermined orientation and spacing between each 2D medical image in the new set of 2D medical images is determined using machine learning techniques.

[0014] • The predetermined orientation and spacing between each 2D medical image in the new set of 2D medical images is user configurable.

[0015] • Automatically detect missing slices from the set of 2D medical images.

[0016] Generate a 2D image corresponding to the missing slice using interpolation techniques.

[0017] • The segmentation technique is based on one or a combination of the following techniques: threshold based, decision tree, chained decision forest and neural network methods.

[0018] • Use voxel-based classification techniques, where voxel information from each axis or plane is considered.

[0019] - Calculating the likelihood that a voxel of the 3D image has a property similar to the patient-specific anatomical feature using a logistic or probability function.

[0020] • A neural network determines the weights for each axis or plane in each voxel of the 3D image.

[0021] • Use multi-channel training to further improve the segmentation technique.

[0022] • Each channel represents a 2D image corresponding to a slice position within the 3D space of the 3D image.

[0023] Use ground truth images to represent channels.

[0024] Generating a 3D mesh model of the patient-specific anatomical features from the segmented 3D image, and generating the 3D printable model from the 3D mesh model.

[0025] The 3D mesh model is further processed using finite element analysis.

[0026] Automatically detect points or areas in the 3D mesh model that require further post-processing steps.

[0027] Further post-processing steps include the placement of pins or other joining structures.

[0028] Automatically determine the optimal placement of pins or other engaging structures.

[0029] - Automatically determine parameters of the patient's anatomical features from analysis of the generated 3D images, such as the volume or size of the anatomical features, thickness of different layers of the anatomical features.

[0030] • The specific anatomical feature is the heart, the measured parameter includes one of: the volume of the heart, the blood volume in each chamber of the heart, the thickness of different layers of the heart wall, the size of a specific blood vessel.

[0031] The 3D printable model is 3D printed as a 3D physical model such that it represents a scale model of the patient-specific anatomical feature, such as a 1:1 scale model or a more appropriate scale model, such as a reduced scale model or an enlarged scale model of the patient-specific anatomical feature depending on the desired downstream application.

[0032] • The 3D printable model is 3D printed wherein key or important features of the particular anatomical feature are made readily visible or accessible.

[0033] - Generating a 3D mesh from the set of 2D medical images, wherein the 3D mesh is a polygonal representation of the volume of the patient-specific anatomical features.

[0034] • Extracting lines or splines from the 3D mesh along the direction of the patient-specific anatomical features.

[0035] • Using a classifier to identify the anatomical features from the extracted lines or splines.

[0036] The method further comprises the step of generating a wireframe model of the 3D mesh.

[0037] • Using a classifier to identify physical properties of the anatomical features from the extracted lines or splines.

[0038] • Using a classifier to identify the pathology of the anatomical feature from the extracted line or spline curve.

[0039] • The classifier is trained to recognize specific anatomical features.

[0040] • The classifier is trained to determine the parameters of the specific anatomical feature, such as its position, size or thickness relative to the human body.

[0041] • The classifier is trained to determine potential defects or pathologies of the specific anatomical feature.

[0042] The classifier is a principal component analysis classifier.

[0043] The method further comprises the step of segmenting the 3D printable model into a set of 3D printable models, wherein the set of 3D printable models comprises connectors, wherein the position of each connector is automatically generated.

[0044] The 3D printable model is determined based on the patient's specific anatomy and pathology.

[0045] • Segmentation of the 3D printable model cannot be determined solely based on surface assessment of the patient's specific anatomy.

[0046] The connector is a magnetic or metal element.

[0047] • Each connector is positioned so that a set of 3D printed physical models from the set of 3D printed models can be connected to represent the patient-specific anatomical features and prevented from being incorrectly connected.

[0048] The set of 3D printed physical models represents a scale model of the patient-specific anatomical features, such as a 1:1 scale model or a more appropriate scale model, such as a reduced scale model or an enlarged scale model of the patient-specific anatomical features depending on the desired downstream application.

[0049] • Making key or important features of the particular anatomical feature readily visible in the set of 3D printable physical models.

[0050] • Making key or important features of the particular anatomical feature readily accessible in the set of 3D printable physical models.

[0051] On the other hand is a 3D physical model representing a scale model of the patient-specific anatomical feature, which is 3D printed from the 3D printable model generated according to the method steps defined above, wherein the scale model is a 1:1 scale model or a more suitable scale model, such as a reduced scale model or an enlarged scale model of the patient-specific anatomical feature depending on the desired downstream application.

[0052] On the other hand is a computer-implemented system for generating a 3D printable model of a patient-specific anatomical feature from a set of 2D medical images, the system comprising a processor for automatically generating a 3D image from the set of 2D medical images, segmenting the generated 3D image using a machine learning-based image segmentation technique, and outputting the 3D printable model of the patient-specific anatomical feature from the segmented 3D image.

[0053] On the other hand is a computer-implemented method for printing 3D models of patient-specific anatomical features, comprising: uploading a set of 2D medical images to a server, processing the set of 2D medical images into a 3D printable model of the patient-specific anatomical features on the server; the server transmitting instructions for printing the 3D printable model to a printer, wherein a security engine verifies that the 3D printable model is associated with correct patient data, and wherein an end user located at a remote location from the printer manages the printing of the 3D printable model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Aspects of the invention will now be described by way of example with reference to the following drawings, each of which illustrates features of the invention:

[0055] Figure 1 A diagram showing the workflow of the Axial3D system.

[0056] Figure 2 A diagram showing the hash of a file illustrating a 3D printable model.

[0057] Figure 3 A set of DICOM stacked images is shown with pixels indicated as boxes.

[0058] Figure 4 A set of DICOM stack images and 3D images are shown with the voxels indicated.

[0059] Figure 5 3D image showing selection in voxel space.

[0060] Figure 6 Displays specific voxels from 3 orthogonal planes.

[0061] Figure 7 Demonstrating data registration of two different datasets for a single patient.

[0062] Figure 8 Show a diagram illustrating isometric slices in a particular plane.

[0063] Fig. 9 Shows a diagram illustrating multi-pass training.

[0064] Fig.10 Shows a diagram illustrating multi-pass training.

[0065] Fig.11 Wireframe models showing meshes used for specific anatomy.

[0066] Fig.12 Diagram showing wireframe model, anatomy, and superimposed model of anatomy and validated wireframe model.

[0067] Fig.13 Showing a 3D bone model with spline curves.

[0068] Fig.14 Three spline curves showing the skeleton.

[0069] Fig.15 A 3D printable model of a heart is shown.

[0070] Fig.16 A 3D physical model showing a heart printed in two parts. DETAILED DESCRIPTION

[0071] This detailed description section describes one embodiment of the invention, referred to as the Axial3D system.

[0072] Figure 1 A diagram illustrating the Axial3D system workflow for ordering a 3D printed model is shown. The Axial3D system uses machine learning-based technology to automatically generate patient-specific 3D anatomical models based on the patient's scan.

[0073] 3D anatomical models can be generated, printed and delivered within 24 to 48 hours.

[0074] like Figure 1 As shown in the example above, 3D printing can be requested through the Axial3D dedicated portal as shown below:

[0075] Register on the Axial3D ordering platform https: / / orders.axial3d.com / (10);

[0076] Select New Print and fill in the patient details such as date of birth, gender, anatomical area of ​​concern, despatch date, anatomical model service required, material type, pathology description, lead consultant details;

[0077] Send a request to the PACS manager or radiologist or upload the DICOM itself (11);

[0078] · Provide 3D annotations or written descriptions upon request. Process the data into 3D printable files by Axial3D software or personnel (12);

[0079] The final print-ready SLT / OBJ files are securely sent via VPN to the on-site 3D printers, each with its own wireless connection (13);

[0080] 3G / 4G network to send and receive data to Axial3D web application;

[0081] If the print order is too large for internal capacity - Axial3D prints and ships the 3D model to the customer.

[0082] If the order is too large or complex, Axial3D printing is managed by Axial3D’s printing service (14).

[0083] As an example, a clinician or radiologist can order a 3D print of a patient-specific anatomical feature via a web portal. The Axial3D system then automates the entire steps of the 3D printing process from processing the 2D medical image to sending instructions to the 3D printer for printing the patient-specific anatomical feature. The clinician is then able to receive the 3D physical model in a timely manner, such as within 24 or 48 hours after the order is placed, with minimal or no involvement on the part of the clinician. The Axial3D system also provides the clinician with additional reports on specific anatomical features, as well as 3D physical models based on detailed analysis of the specific anatomical features.

[0084] Cybersecurity Processes in Medical 3D Printing

[0085] We have developed a digital platform to safely and verifiably produce and deliver 3D printed anatomical models on demand, globally, at scale, and in a wide range of scenarios: making it applicable not only to health authorities, private hospitals, and surgical suites, but ultimately to any hospital. The technical challenge is to provide indisputable verification of the provenance of both the virtual models generated from the patient’s anonymized data and any physical models 3D printed from that data. The stakeholders involved in this process represent multiple parties spread across multiple organizations, so they need to be reliably identified, authenticated, and able to independently verify the provenance of these models.

[0086] This enables remote printing of 3D anatomical models, where printing is done in one location and remotely controlled at another location. Once a 3D physical model is ordered, a 3D model is generated from a 2D medical scan and then remotely reviewed, approved and controlled by a 3D printing technician.

[0087] 3D printing technicians can also remotely control more than one printer, and the system can automatically decide how to best select or arrange printing on one or more printers.

[0088] In order to prove or verify that the printed 3D physical object is the 3D physical object sent remotely and that it is associated with the correct patient without exposing any confidential patient data, a cybersecurity process is critical.

[0089] Cryptographic signature of files

[0090] We create and store a hash of the 3D model file representing the 3D printable model of a specific anatomical feature and use the hash to recreate the object or 3D physical model anytime we need it. This hash can be used to quickly determine if the file has been modified.

[0091] Every time we upload or change a file on the web application we need to create a new hash, however the hash created at the end of the process is the canonical hash of the printed file. Therefore, all previous files are quality controlled 'drafts'. The specification is what we publish so that the user has the final file.

[0092] In the process of generating an anatomical model from a medical scan, the data undergoes several transformations and modifications. To record these changes, a hash file is generated at each of these steps. The process of identifying the anatomy in the scan produces a label on the scan, which is then used to generate the print file. The hashing process records this and serves as a history of the changes. Modifications to the file are stored and used to provide tracking of the file's provenance. In this way, users can be guaranteed the provenance of the files they use.

[0093] We have implemented a system that allows for the cryptographic signing of files and their subsequent distribution. The distribution of files for printing is managed by providing a decentralized file signing service. This is achieved by cryptographically signing the files using private / public key based encryption. This allows remote parties to verify the files in a secure manner.

[0094] Provide a service that allows the download of a file and any subsequent testing of the file's correctness. The file can be stored on an object file system (such as S3) along with a hash of the file. A 'central' repository of hashes then links the file to the order. This repository can be a file, a database, or a distributed ledger.

[0095] Figure 2 A diagram illustrating tracking of file modifications is shown, where changes to files are committed to a repository, and changes to instances of the repository are synchronized between repositories.

[0096] Our system ensures that only authenticated files can be printed. Files are signed and only files that pass a cryptographic challenge are accepted for printing. As a result, only files that have been signed and authenticated by the authentication server can be sent to the printer. This also means that all files are encrypted at rest and in transit, and modifications can be logged and observed without viewing the contents of the file.

[0097] Our system can sit in front of the printer and ensure only encrypted files are sent for printing. Files can be decrypted in transit when printing is complete, ensuring an encrypted version of the file is always stored / transmitted.

[0098] Work natively in 3D space

[0099] Most segmentation methods apply algorithms to 2D images and then generate 3D models from the segmented 2D images.

[0100] Figure 3 A set of DICOM stacked images is shown with pixels indicated as boxes. Figure 4 A set of DICOM stacked images (40) and a 3D image (41) with indicated voxels are shown. Figure 3 The scan slices shown in the figure are transformed into Figure 4 Instead of a single volume as shown in , we work natively in 3D space. We compute the zero point of our volume coordinate system and orient all slices in the scan to this. This allows us to compute the alignment of the slices with reference to the volume, and observe properties of the group of slices. This means that we are now working natively in 3D with our 3D feature detection filters, which essentially becomes a voxel classification instead of a pixel-based classifier.

[0101] exist Figure 5 A key advantage of working natively in 3D is that the system incorporates orthogonal information in the scoring metric. This is done by taking into account Figure 6 The simplest indication is where 3 planes are considered together. Showing a particular voxel with three plane cuts through it reveals more information about the likelihood of the voxel being a member of a particular class. By incorporating information from all planes for each voxel, connections between bones or other anatomical segments can be more efficiently identified. This is because transformations in the image (e.g., voxel intensities) are more easily spotted when all planes are considered. The result is that all voxels in a scan can be used to train the algorithm simultaneously. In practice, this means that larger intervals and biological features can be encoded in the algorithm to overcome specific challenges at anatomical intersections, such as the myocardial wall to the ventricle (heart) or bone joints, for example Figure 5 Those shown in .

[0102] Combine image registration and perform multimodal reasoning

[0103] We can register multiple image stacks and modality (e.g. MRI & CT or MRI and MRI, with different structures highlighted in more detail) scans to cover voxels from different scans, e.g. Figure 7, which illustrates the data registration of two different datasets of a single patient. We can identify feature points in the scans to help map pixels from one scan to another. A feature point is a point or shape that is shared between individuals through common ancestry. It can be biologically meaningful, such as the shape of the canthus of the skull, or mathematically represented as the point of highest curvature on the surface of a bone. This means that information from multiple scans can be used simultaneously to identify features for a machine learning algorithm. Since both modalities can be considered different perspectives of the same anatomy, the combination allows us to add additional information to the training phase. In this way, 2D medical images, such as those provided by CT, MRI or PET scans, can be processed together.

[0104] The Axial3D system includes the following steps: (i) receiving a 2D medical image, (ii) automatically generating a 3D image from the 2D medical image, and (iii) processing the 3D image to segment or classify the 3D image. A 3D printable model can then be generated from the segmented 3D image.

[0105] 3D image data file formats include, for example, any point cloud format or any other 3D imaging format.

[0106] Key features of the system are, but are not limited to, the following:

[0107] Since 3D images contain image data from more than one direction, anatomical transformations are easier to identify. By comparison, when processing slices of 2D images, only information from one direction is available.

[0108] • Therefore, this also improves the identification of specific anatomy.

[0109] Combining information from multiple planes

[0110] To image a specific anatomy, cross-sectional images are taken at any angle. As an example, an MRI scan of the heart acquires 2D images in different directions. Working locally in 3D improves the accuracy of the generated 3D printable model (as measured using standard metrics such as, but not limited to, the DICE coefficient). On a per-voxel basis, the accuracy of the prediction is improved by considering the 3D properties of the voxel rather than the 2D properties of the pixel and combining them. Each plane or 2D image and its constituent pixels become features of the 3D volume. For each voxel, the features of each pixel are encoded as features of the voxel. The network is then trained and the appropriate weight to be given to each plane is determined. Each feature is represented as a discrete range and optimized by the neural network training process. In this way, the training process can learn the appropriate integration of additional information from all planes.

[0111] Post-segmentation utility for anatomical feature delineation

[0112] When a piece of anatomy is fully and accurately segmented, it is possible to measure several physical properties of the anatomy (eg, the heart). The segmented anatomy can be measured by relating pixel size to a physical scale from a coordinate system.

[0113] Determine parameters of anatomical features such as, but not limited to:

[0114] The volume of the anatomical region;

[0115] The volume of a cavity of an anatomical feature, such as the blood in each chamber of the heart;

[0116] The thickness of different layers of anatomical features (e.g., heart wall);

[0117] The size and diameter of features (such as blood vessels or bones);

[0118] Directional properties of the shape, for example in the case of scoliosis - detection of scoliosis and type of scoliosis, measuring or determining the angle or degree of the curve;

[0119] Cortical bone density - determines whether a screw can be fitted, or automatically determines the parameters of a screw that fits;

[0120] Aneurysms – whether the coil or clip is able to stop or block blood flow;

[0121] The force required to break a bone;

[0122] Further information on the pathology or extent of the injury.

[0123] Information about additional pathology not reported by the clinician, such as the location of a previously unknown fracture.

[0124] When a 3D printable model is ordered, the system generates and sends a report with the above information to the physician. This can improve the surgeon's preoperative planning and further reduce costs for healthcare providers. For example, by more accurately understanding the size of the vessel, the surgeon can make an informed choice about the appropriate stent size before surgery. The system can also automatically determine the parameters of the stent.

[0125] Automatically identify and repair spatial errors and inconsistencies in volumetric data

[0126] Figure 8A diagram illustrating equidistant slices in a particular plane is shown. We apply two methods, one for identifying nonlinear slices and the other for identifying missing slices. Trajectories of 2D slices are plotted and analyzed. If a slice is found to be above or below a certain trajectory threshold, it is removed from the analysis before generating the 3D image. The slices must be in planes that are congruent to each other, they appear parallel, and are equidistant from a reference plane. Assuming that there is a set of isolinear slices, we can identify such a set of slices and such a set of slices minus a slice. Therefore, we are tolerant in identifying an isolinear slice as one slice.

[0127] Interpolated data of multiple slices including slices from multiple angles are combined.

[0128] Next, we developed a method for inferring missing data between two slices. This relies on the ability to create missing slices using the correct 3D geometry and interpolated pixel values.

[0129] Many medical imaging datasets contain slices from a patient at multiple angles. CT scans are often limited in their ability to acquire slices at standard angles, whereas MR scans often require oblique scanning. Oblique scanning is often used in MR imaging to minimize the total number of images to be collected and, therefore, to reduce the time and cost of performing the scan. Typically, when this technique is used, a relatively small number of slices are acquired at each oblique angle (typically 5 to 10 images) with a large slice interval (5 to 10 mm); oblique scanning is usually performed in three nearly perpendicular directions (axial, coronal, sagittal) and an additional oblique axis, however, the imaging angle and number of scans are left to the discretion of the medical practitioner.

[0130] As a result, too few slices are available along a single axis to generate a complete volume of sufficient quality. For example, the spacing between each slice may be greater than 5 mm, thereby completely missing important anatomical features.

[0131] When viewed in combination, the resulting images can provide only enough visual information about a particular lesion: each part of the lesion may be located in a large gap in one of the scans, while it may be visible in the other scans. For example, a 10 mm tumor mass is visible in only one slice of the axial scan, one slice of the coronal scan, and two slices of the sagittal scan; in this scenario, the oncologist will view these four images simultaneously to gain a three-dimensional understanding of the tumor shape and volume.

[0132] The Axial3D system is able to automatically decide how to process a 2D medical image in order to provide an accurate 3D physical print of a specific anatomical feature, wherein any critical or important features of the specific anatomical feature are made visible. These critical or important features can also be made easily accessible by segmenting the 3D physical model into a set of connectable 3D physical models. These processing decisions can be based on specific anatomical features, specific pathologies, or any other pre-configured or learned parameters. This in turn aids in the diagnosis and treatment of patients, thereby improving surgical planning and patient care.

[0133] In this way, we show how to interpolate multiple simultaneous stacks into one volume. This fully exploits intersecting slices to achieve higher information density and creates a high-fidelity interpolation. The slice spacing of the reconstructed volume is limited by the original tilt scan spacing: depending on the number of tilt scans (typically 3 or 4, as mentioned above), the slice spacing of the reconstructed volume can be as low as one-fifth of the original scan (e.g., if the tilt scan slice spacing varies between 5mm and 6mm, then the reconstructed volume spacing can be as low as 1mm).

[0134] Interpolation is accomplished by finding the absolute position of the corners of each DICOM image in each stack relative to the origin determined by the scanner itself and reported in the DICOM header. This allows a bounding box to be constructed to encompass all images in the space in which they are embedded. By discretizing the bounding box so that it represents a volume of voxels spanning the size of all stacks, a mapping from the space of each stack of DICOM to the space of the new volume can be determined. At each point in the new volume, the pixels K closest to the point in DICOM are determined, and their distance d is calculated. At this point, the voxel value M is calculated as a weighted sum:

[0135] where q = d -1

[0136] For each imaging orientation, a stack of images is given as part of the original data set, and for each orientation there are 20 to 30 such stacks, representing scans taken at the same location but at different times. Each interpolation is generated for a series of DICOM images across all scan orientations but for one timestamp.

[0137] This facilitates three-dimensional interpolation. Therefore, prior to the generation of a 3D image, the original 2D slices from multiple angles are transformed into a set of evenly distributed parallel 2D slices.

[0138] Multi-channel training

[0139] Here we describe improving the segmentation network by adding "up and down" slices alongside a typical input image. This informs the neural network about continuous structures and those that are just artifacts of a particular scan. We expect improvements in the neural network in particular in correctly identifying thinner bone fibers while removing areas of the image that have similar Hounsfield values ​​but do not belong to the same anatomical class. For the three-channel example, the neural network requires an input of the following shape:

[0140] (batch_size, channels, X, Y)

[0141] The data is segmented to fit the required memory size. The segmented data can then be fed into any neural network or any image analysis platform.

[0142] To achieve this, each stack is first filled with "images of zeros" at the top and bottom of the stack. This means that groups of three slices can be formed into objects with the same total number of input objects, such as Fig. 9 As shown in .

[0143] Each input triplet will have a ground truth or gold standard corresponding to the ground truth image associated with the center image to give "above and below" information, such as Fig.10 As shown in . When additional channels are added, each image and ground truth pair will still exist. The same principle applies to any number of odd channels; for every two channels, another layer of padding should be added to keep the same number of inputs. When dealing with an even number of channels, the situation is slightly more complicated, but this is less desirable because it removes the good aspect of symmetry. In practice, it may also be more useful to add padding corresponding to the minimum Hounsfield value of the stack, because this avoids very strong transformations that may hinder learning. In the case where an image has a padded image above or below it, there is only less information available to make predictions, and the presence of padding should not affect the predictions themselves.

[0144] Examples of extracted 3D features are as follows:

[0145] Conversion;

[0146] Pixel intensity

[0147] ·shape;

[0148] 3D shapes;

[0149] "Wireframe" shape detection

[0150] We generate an isosurface of the anatomical feature by transforming the probability distribution matrix in the inference algorithm into a discrete scalar volume. Next, this isosurface is used to generate a volume mesh, which is a polygonal representation of the volume of the anatomical feature. After generating the surface, we draw a wireframe representation of the surface. This wireframe representation consists of a series of spline curves that form the outline of the given surface mesh. These can be compared to the existing mesh outlines to see if they match.

[0151] Fig.11 Wireframe models showing meshes used for specific anatomy.

[0152] Fig.12 Diagram showing wireframe model, anatomy, and superimposed model of anatomy and validated wireframe model.

[0153] Building a wireframe model of the mesh helps quickly identify specific shapes and their location relative to the body. This in turn improves the accuracy of the 3D printable model and the 3D printed physical model.

[0154] Examining lines in one dimension to compare shapes is less computationally intensive than examining 3D surfaces to compare shapes. Additionally, examining continuous lines helps identify continuous anatomy, whereas examining 3D surfaces is more prone to error.

[0155] A simple method for determining bone class. Lines can be drawn along the surface of the anatomy, which provide unique identifiers of feature points on the surface of the anatomy. ML models can be trained to recognize sets of peaks and valleys in the surface lines and the relationships between them, which allows these surface lines to be classified and therefore the anatomy to be identified.

[0156] A wireframe representation of the mesh. A single line can be drawn along the length of each bone in the scene to form a spline, such as Fig.13 As shown in , where the lines (131) from the wireframe model show the spline curves of the bones.

[0157] Fig.14 Displays the spline curve of the first bone, the spline curve of the second bone, and the spline curve of the third bone.

[0158] The splines above show two different bones - splines 2 and 3 are the same bone in different people. A classifier can be trained to recognize the difference between the two splines. The classifier can include a PCA (Principal Component Analysis) classifier.

[0159] Directional fixation

[0160] Detect overlaps in shapes / volumes / meshes.

[0161] Allows comparison between these shapes in 3D space.

[0162] Register the images to the origin of the DICOMs from two different scans (e.g. one MRI and one CT). This can be done in several ways; if both sets of images are referenced to the same origin, then the scans can simply be overlaid. However, if these do not exist, the algorithm will detect the anatomy in both scans and use 3D object retrieval techniques to overlay the anatomy and identify the same parts in both scans. These can be combined with conventional techniques from 2D registration to provide a higher level of confidence.

[0163] Automatic detection of pin placement and other post-processing steps

[0164] We perform shape modeling whereby we determine the weakest and strongest locations on the mesh. This can be done by bending and distorting the mesh and determining the points of maximum and minimum bending. The output of this stage will be a heat map of the mesh which provides a score of how strong the mesh is at a given point. This allows us to identify areas that need strengthening. It will also allow us to detect places that could be used for the placement of magnetic connectors.

[0165] We have developed an algorithm that allows us to identify articulation points in a 3D mesh. We use this to determine where additional support structures should be applied to the model. We apply uniform vertical pressure to the mesh and identify the degree of rotation of the polygons after the pressure is applied. Points or polygons that are rotated 90 degrees or more are most in need of further reinforcement. Finite element analysis can be applied to the 3D mesh to develop a mesh map that captures the structural properties of the mesh. This information can be used to detect locations on the mesh that can be used to deploy pins and other joint structures.

[0166] We have implemented a heuristic algorithm that allows us to efficiently enumerate potential solutions to the problem and identify the most suitable solution. We have defined placement criteria for locating pins that are support structures between multiple parts of the model. We then use these as rules to optimize the placement of such support structures. We employ wave functions to identify and optimize the placement of pins and other structures in the 3D mesh. These are then solved by wave function folding to obtain the optimal position of the pins. Additional constraints can be imposed on the solution to avoid specific features identified by the user.

[0167] Another use case is where we split a model into two or more pieces and want to reconnect them using magnets. We have developed an algorithm that allows us to identify the best placement of these attachment magnets. This is an extension of the above algorithm whereby we add further constraints on the twisting, squeezing or twisting of the model that capture the properties of the magnets.

[0168] Deconstructed anatomy with magnetic connectors

[0169] The user defines a split line through the entire model, or splits the model with a non-uniform cut to separate a specific piece of anatomy (e.g., pubic and ilium from ischium within a hemi-pelvis). The user then enters the diameter and depth of the magnet, and the software automatically indents the magnet into the surface of the anatomy, or, if the wall is too thin, incorporates a cylindrical inset on the outside of the model (Embedded Cylindrical Inset Model below).

[0170] The parts are segmented so that it is not possible to connect the different parts together in the wrong way. Magnetic or metal elements are placed to guide the different parts together. The metal element is magnetically attracted to the element located on the other part, making it impossible to connect the different parts in the wrong way.

[0171] As an example, Fig.15 A 3D printable model of a heart is shown. Fig.16 A 3D physical model of a heart is shown printed in two separate parts. This allows the physician to see the 3D printed physical anatomy as a whole, while being able to open it up and see what is inside. The printed model can then be put together again, knowing that it has been put together in the correct way.

[0172] Different parts can be printed in different colors or with different material formulations (ie, soft and hard polymers).

[0173] Key Features

[0174] This section summarizes the most important high-level features; embodiments of the present invention may include one or more of these high-level features, or any combination of any of these features. It should be noted that each feature may therefore be an independent invention and can be combined with any one or more other features.

[0175] We group these features into the following categories:

[0176] A. Working natively in 3D

[0177] B. Wireframe Model

[0178] C. Split the 3D printable model into a group of 3D printable models

[0179] D. Remote printing

[0180] A. Working natively in 3D

[0181] A computer-implemented method for generating a 3D printable model of a patient-specific anatomical feature from a 2D medical image, wherein:

[0182] (a) Automatically generate 3D images from a set of 2D medical images

[0183] (b) segmenting the generated 3D image using an image segmentation technique based on machine learning; and

[0184] (c) creating a 3D printable model of the patient-specific anatomical features from the segmented 3D image.

[0185] Optional:

[0186] • The set of 2D medical images are images of the patient acquired from one or a combination of: CT, MRI, PET and / or SPCET scanners.

[0187] Simultaneously process 2D medical images from multiple scanning technologies.

[0188] - Automatically pre-processing the set of 2D medical images such that important or critical features of the patient-specific anatomical features are made visible in the 3D printable model.

[0189] • Pre-processing of the 2D medical images is based on the specific anatomical features, the patient's specific pathology or any downstream application, such as pre-operative planning or training purposes.

[0190] The pre-processing of the 2D medical images is based on the specific anatomical features, the specific pathology of the patient.

[0191] The set of 2D medical images is pre-processed to generate a set of new 2D medical images uniformly distributed according to a predetermined orientation.

[0192] • The predetermined orientation is based on the patient specific anatomical features, the patient specific pathology or any downstream application, such as pre-operative planning or training purposes.

[0193] • The predetermined orientation and spacing between each 2D medical image in the new set of 2D medical images is determined using machine learning techniques.

[0194] • The predetermined orientation and spacing between each 2D medical image in the new set of 2D medical images is user configurable.

[0195] • wherein missing slices from the set of 2D medical images are automatically detected.

[0196] - Correcting the missing slices by generating images corresponding to the missing slices using interpolation techniques.

[0197] • The segmentation technique is based on one or a combination of the following techniques: threshold-based, decision tree, chained decision forest, and neural network methods.

[0198] • Use voxel-based classification techniques, where voxel information from each axis or plane is considered.

[0199] - Calculating the likelihood that a voxel of the 3D image has a property similar to the patient-specific anatomical feature using a logistic or probability function.

[0200] • A neural network determines the weights for each axis or plane in each voxel of the 3D image.

[0201] • Use multi-channel training to further improve the segmentation technique.

[0202] • wherein each channel represents a 2D image corresponding to a slice location within the 3D space of the 3D image.

[0203] Generate a 3D mesh model of the patient-specific anatomical features from the segmented 3D image.

[0204] The 3D mesh model is further processed using finite element analysis.

[0205] Automatically detect points or areas in the 3D mesh model that require further post-processing steps.

[0206] Further post-processing steps include the placement of pins or other joining structures.

[0207] • Wherein the optimum placement of pins or other engaging structures is automatically determined.

[0208] The 3D printable model is based on the generated 3D mesh model.

[0209] The 3D printable model is 3D printed as a 3D physical model such that it represents a scale model of the patient-specific anatomical feature, such as a 1:1 scale model or a more appropriate scale model, such as a reduced scale model or an enlarged scale model of the patient-specific anatomical feature depending on the desired downstream application.

[0210] • The 3D printable model is 3D printed wherein key or important features of the particular anatomical feature are made readily visible or accessible.

[0211] - Automatically determine parameters of the patient's anatomical features from analysis of the generated 3D images, such as the volume or size of the anatomical features, thickness of different layers of the anatomical features.

[0212] • The specific anatomical feature is the heart and the parameters measured are: the volume of the heart, the volume of blood in each chamber of the heart, the thickness of the different layers of the heart wall, the size of specific blood vessels.

[0213] B. Wireframe Model

[0214] A computer-implemented method for identifying anatomical features from a set of 2D medical images, the method comprising:

[0215] (a) generating a 3D mesh from the set of 2D medical images, wherein the 3D mesh is a polygonal representation of the volume of the anatomical feature;

[0216] (b) extracting lines or splines from the 3D mesh along the direction of the anatomical feature; and

[0217] (c) Using a classifier to identify anatomical features from the extracted lines or spline curves.

[0218] Optional:

[0219] • Using the classifier to identify physical properties of the anatomical feature from the extracted lines or splines.

[0220] • Using the classifier to identify pathology of the anatomical feature from the extracted line or spline curve.

[0221] The method further comprises the step of generating a wireframe model of the 3D mesh.

[0222] • Automatically generating a 3D image from said set of 2D medical images and generating said 3D mesh from said segmentation of said 3D image.

[0223] • The classifier is trained to recognize specific anatomical features.

[0224] • The classifier is trained to determine the parameters of the specific anatomical feature, such as its position, size or thickness relative to the human body.

[0225] • The classifier is trained to determine potential defects or pathologies of the specific anatomical feature.

[0226] The classifier is a principal component analysis classifier.

[0227] C. Split the 3D printable model into a group of 3D printable models

[0228] A computer-implemented method for segmenting a 3D printable model of a patient-specific anatomical feature into a set of 3D printable models, wherein the method includes the step of automatically segmenting the 3D printable model into a set of 3D printable models, wherein the 3D printable models include connectors, wherein the position of each connector is automatically generated.

[0229] Optional:

[0230] The segmentation of the 3D printable model is determined based on the patient's pathology and anatomy. Thus, no information can be obtained by evaluating the surface of a given structure.

[0231] The connector is a magnetic or metal element.

[0232] • Each connector is positioned so that a set of 3D printed physical models from the set of 3D printed models can be connected to represent the patient-specific anatomical features and prevented from being incorrectly connected.

[0233] The set of 3D printed physical models represents a scale model of the patient-specific anatomical features, such as a 1:1 scale model or a more appropriate scale model, such as a reduced scale model or an enlarged scale model of the patient-specific anatomical features depending on the desired downstream application.

[0234] • Making key or important features of the particular anatomical feature readily visible in the set of 3D printable physical models.

[0235] • Making key or important features of the particular anatomical feature readily accessible in the set of 3D printable physical models.

[0236] D. Remote printing

[0237] A computer-implemented method for printing a 3D model of a patient-specific anatomical feature, comprising:

[0238] (a) Upload 2D medical images to the server,

[0239] (b) processing the 2D medical image into a 3D printable model of the patient-specific anatomical features on the server; and

[0240] (c) the server transmits instructions for printing the 3D printable model to a printer, wherein the security engine verifies that the 3D printable model is associated with the correct patient data,

[0241] Wherein an end user located at a remote location from the printer manages the printing of the 3D printable model.

[0242] Optional:

[0243] • The 2D medical images and additional metadata sets are anonymized before being sent to the server so that no identifiable healthcare or personal information is transmitted to the server.

[0244] The end user remotely schedules, initiates or approves printing of a 3D printable model on one or more printers via a web application.

[0245] The end user remotely controls one or more printers, and the printing is automatically arranged on the one or more printers.

[0246] A hash of the file corresponding to the 3D printable model file is created and stored in a central repository.

[0247] The central repository is accessed by the server, wherein the central repository is a file, a database or a distributed ledger.

[0248] The hash is used to recreate or authenticate the print or any subsequent print of the 3D patient-specific anatomical features.

[0249] - Modifications to the file are stored with the hash and used to provide an audit trail of the file's provenance.

[0250] The hash is used to determine if the file has been modified.

[0251] • The distribution of one or more files for 3D printing one or more specific anatomical features is managed by a centralized file signing service.

[0252] The file corresponding to the 3D printable model is encrypted using private / public key based encryption.

[0253] The security engine ensures that only encrypted files are transferred for printing.

[0254] · Files are only decrypted during transmission when printing is complete.

[0255] Notice

[0256] It should be understood that the above referenced arrangements are merely illustrative of the application of the principles of the present invention. Many modifications and alternative arrangements may be devised without departing from the spirit and scope of the present invention. Although the present invention has been illustrated in the accompanying drawings and fully described above with particularity and detail in connection with what are currently considered to be the most practical and preferred embodiments of the present invention, it will be apparent to those of ordinary skill in the art that many modifications may be made without departing from the principles and concepts of the present invention as set forth herein.

Claims

1. A computer-implemented method for generating a 3D printable model of a patient-specific anatomical feature from a 2D medical image, wherein the 2D medical image is a cross-section taken at any angle; wherein the neural network is configured to infer a 3D image from the 2D medical image such that, for each voxel of the 3D image, a feature of each constituent pixel of the 2D medical image is encoded as a feature of the voxel; wherein the neural network has been trained to determine the appropriate weight to be given to each 2D medical image; wherein the neural network is configured to segment the patient-specific anatomical features from the 3D image; and Wherein a 3D printable model of the patient-specific anatomical features is created from the segmented 3D image.

2. The method according to claim 1, wherein a 3D mesh is generated by transforming a probability distribution matrix in an inference algorithm into a discrete scalar volume, wherein the 3D mesh is a polygonal representation of the volume of the patient-specific anatomical feature; wherein lines or splines are extracted from the 3D mesh along the direction of the patient-specific anatomical features; and Wherein the 3D printable model of the patient-specific anatomical feature is created from the 3D mesh.

3. The method according to claim 1 or 2, wherein the set of 2D medical images is preprocessed to generate a set of new 2D medical images uniformly distributed according to a predetermined orientation; and Wherein the 3D image is inferred from the new set of 2D medical images.

4. The method of claim 3, wherein the set of 2D medical images are images of a patient acquired from one or a combination of: CT, MRI, PET and / or SPCET scanners.

5. The method of claim 1, wherein the neural network has been trained to infer the 3D image from a 2D medical image based on at least one of a plurality of scanning techniques.

6. The method of claim 5, wherein the plurality of scanning techniques comprises CT, MRI and / or PET.

7. The method of claim 3, wherein the set of 2D medical images is automatically pre-processed such that important or critical features of the patient-specific anatomical features are made visible in the 3D printable model.

8. The method according to claim 7, wherein the pre-processing of the 2D medical image is based on the specific anatomical features, the specific pathology of the patient or any downstream application.

9. The method according to claim 8, wherein the pre-processing of the 2D medical images is based on pre-operative planning or training purposes.

10. The method of claim 3, wherein the predetermined orientation is based on the patient-specific anatomical features, the patient-specific pathology, or any downstream application; and / or Wherein the predetermined orientation and spacing between each 2D medical image in the new set of 2D medical images is determined using machine learning techniques and / or is user configurable. The method of claim 10 , wherein the predetermined orientation is based on preoperative planning or training purposes.

12. The method of claim 3, wherein missing slices from the set of 2D medical images are automatically detected.

13. The method of claim 12, wherein the 2D image corresponding to the missing slice is generated using an interpolation technique.

14. The method of claim 1 or claim 2, wherein segmenting the patient-specific anatomical features from the 3D image is achieved based on one or a combination of the following techniques: a threshold-based decision tree, a chained decision forest, and a neural network approach.

15. The method of claim 1 or claim 2, wherein a voxel-based classification technique is used, wherein voxel information from each axial or 2D medical image is considered.

16. The method of claim 1 or claim 2, wherein the likelihood that a voxel of the 3D image has a property similar to the patient-specific anatomical feature is calculated using a logistic or probability function.

17. The method of claim 1 or claim 2, wherein the neural network is trained using multi-channel training based on additional "up and down" 2D images as input.

18. The method of claim 3, wherein the set of 2D medical images comprises images taken at different angles; and Wherein before generating the 3D image, the set of 2D medical images are transformed into the set of new 2D medical images in uniformly distributed parallel 2D images via interpolation.

19. The method of claim 1 or claim 2, wherein a 3D mesh model of the patient-specific anatomical feature is generated from the segmented 3D image, and the 3D printable model is generated from the 3D mesh model.

20. The method of claim 19, wherein the 3D mesh model is further processed using finite element analysis.

21. The method of claim 19, wherein points or areas in the 3D mesh model requiring further post-processing steps are automatically detected.

22. A method according to claim 1 or claim 2, wherein the further post-processing step comprises the placement of pins or other engaging structures.

23. The method of claim 1 or claim 2, wherein parameters of the patient anatomical features are automatically determined from analysis of the generated 3D images.

24. The method of claim 23, wherein analyzing the generated 3D image comprises analyzing the volume or size of the anatomical feature, the thickness of different layers of the anatomical feature.

25. The method of claim 23, wherein the specific anatomical feature is a heart and the determined parameters include one of: the volume of the heart, the blood volume in each chamber of the heart, the thickness of different layers of the heart wall, the size of a specific blood vessel.

26. The method of claim 2, wherein a classifier is used to identify the anatomical feature from the extracted line or spline curve, to identify a physical property of the anatomical feature from the extracted line or spline curve, and / or to identify a pathology of the anatomical feature from the extracted line or spline curve.

27. The method of claim 26, wherein the classifier is trained to identify specific anatomical features, to determine parameters of the specific anatomical features, and / or to determine potential defects or pathologies of the specific anatomical features.

28. The method of claim 27, wherein the parameters of the specific anatomical feature include at least a position, a size, or a thickness of the specific anatomical feature relative to a human body.

29. A computer-implemented system for generating a 3D printable model of a patient-specific anatomical feature from a 2D medical image, the system comprising a processor configured to: receiving the 2D medical image, wherein the 2D medical image is a cross-section taken at any angle; inferring a 3D image from the 2D medical image using a neural network such that, for each voxel of the 3D image, features of each constituent pixel of the 2D medical image are encoded as features of the voxel, wherein the neural network has been trained to determine an appropriate weight to be given to each 2D medical image; segmenting the patient-specific anatomical features from the 3D image using the neural network; and A 3D printable model of the patient-specific anatomical features is created from the segmented 3D image.

Images