Systems and methods for automatically segmenting patient-specific anatomical structures for pathology-specific measurements

Abstract

The present disclosure provides systems and methods for multi-modal analysis of patient-specific anatomical features from medical images. The system can receive a medical image of a patient and metadata associated with the medical image indicative of a selected pathology, and automatically classify the medical image using a segmentation algorithm. The system can use an anatomical feature identification algorithm to identify one or more patient-specific anatomical features within the medical image by exploring an anatomical knowledge dataset. A 3D surface mesh model representing the one or more classified patient-specific anatomical features can be generated, such that information can be extracted from the 3D surface mesh model based on the selected pathology. Physiological information associated with the selected pathology for the 3D surface mesh model can be generated based on the extracted information.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority benefit of GB Patent Application Serial No. 2101908.8, filed February 11, 2021, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure is directed to systems and methods for multi-modal analysis of patient-specific anatomical features from medical images to make pathology-specific measurements in diagnosis, planning, and treatment for specific use cases. BACKGROUND

[0004] Creating accurate 3D models of specific portions of a patient’s anatomy aids in changing surgical procedures by providing clinicians with insight to preoperative planning. Benefits include, for example, better clinical outcomes for patients, reduced surgical time and cost, and the ability for patients to better understand the planned surgery.

[0005] However, there remains a need to provide 3D models to provide a deeper understanding of a patient’s anatomy or pathology.

[0006] In view of the foregoing shortcomings of previously known systems and methods, there is a need for enhanced systems and methods for analyzing medical images of a patient to create 3D models to aid in diagnosis, planning, and / or treatment. SUMMARY

[0007] The present disclosure overcomes the shortcomings of previously known systems and methods by providing systems and methods for multi-modal analysis of patient-specific anatomical features from medical images to make pathology-specific measurements in diagnosis, planning, and / or treatment for specific use cases.

[0008] Producing a scaled virtual replica of a patient’s anatomy (e.g., a 3D anatomical model) is an extremely useful tool that can be used to drive personalized patient-specific decisions in clinical practice, such as for preoperative planning. The present disclosure demonstrates how to produce patient-specific 3D models of a patient’s complete anatomy, for example, by constructing machine learning models to automatically detect and segment anatomical structures from medical scans. These models can be trained using a curated, semantically labeled dataset. To produce a 3D segmentation, a neural network or machine learning algorithm is trained to recognize anatomical features within a set of medical images. These images are semantically labeled with the location and landmarks of anatomical features and their constituent parts. Thus, the segmentation algorithm can take a new dataset and its complementary landmarks and use it to identify new anatomical features or landmarks.

[0009] The segmentation process is the first step in generating patient-specific knowledge of anatomical features, which powers decisions in the clinical setting. Technology available from Axial Medical Printing Limited of Belfast, UK, turns 2D medical scans into scaled 3D models of patient anatomy, allowing for 3D decision making and understanding. The output of the segmentation process is a set of precise coordinates representing anatomical features in the scan. This representation of the anatomy allows for explicit statements about the features to be made, e.g., standard measurements such as size, length, volume, diameter, oblique cross-section, etc. Thus, a surgeon can calculate the shape and location of anatomical features or pathology and incorporate it into a personalized decision making process. These measurements can be used to drive key decisions about the patient's condition and any proposed intervention.

[0010] More importantly, the system described herein can distinguish between the normal and pathological states of anatomical structures and any anatomical features. The training process can be further refined with this information, and this information can be used to drive further classification of anatomical features. For example, blood can be identified and segmented in a medical scan. By incorporating information about the pathological state, blood clots within the blood vessels can also be identified and segmented, such that the type and severity of the pathology can be identified. In combination with the measurement data about the anatomy, this information is critical to making decisions in acute clot-based pathologies, such as stroke or coronary heart disease.

[0011] A pathology-specific patentable artifact can be created by combining the automatic segmentation algorithm described herein with large labeled training datasets specific to each pathology, such that the combination of the appropriate algorithm and specific data creates a unique artifact set for each pathology. The ability to provide specific functional groupings of segmentation provides significant benefits for specific clinical problems. Furthermore, the ability to provide automatic segmentation also opens up many pathology-specific applications that would benefit from the system described herein.

[0012] According to one aspect, a method for multi-modal analysis of patient-specific anatomical features from medical images is provided. The method can include receiving, by a server, a medical image of a patient and metadata associated with the medical image indicative of a selected pathology; automatically processing, by the server, the medical image using a segmentation algorithm to label pixels of the medical image and generate scores indicative of a likelihood that the pixels are correctly labeled; probabilistically matching, by the server, associated groups of the labeled pixels to an anatomical knowledge dataset using an anatomical feature recognition algorithm to classify one or more patient-specific anatomical features within the medical image; generating, by the server, a 3D surface mesh model that bounds a surface of the one or more classified patient-specific anatomical features; extracting, by the server, information from the 3D surface mesh model based on the selected pathology; and generating, by the server, physiological information of the 3D surface mesh model associated with the selected pathology based on the extracted information. For example, the information extracted from the 3D surface mesh model can include a 3D surface mesh model of an anatomical feature isolated from the one or more classified patient-specific anatomical features based on the selected pathology.

[0013] Generating, by the server, physiological information of the 3D surface mesh model associated with the selected pathology can include determining a start point and an end point of the isolated anatomical feature; taking slices at predefined intervals along an axis from the start point to the end point; calculating a cross-sectional area of each slice bounded by a perimeter of the isolated anatomical feature; extrapolating a 3D volume between adjacent slices based on the respective cross-sectional areas; and calculating a total 3D volume of the isolated anatomical feature based on the extrapolated 3D volumes between adjacent slices.

[0014] Generating, by the server, physiological information of the 3D surface mesh model associated with the selected pathology can include determining a start point and an end point of the isolated anatomical feature and a direction of travel from the start point to the end point; projecting rays at predefined intervals along an axis in at least three directions perpendicular to the direction of travel and determining a distance between an intersection of each projected ray and the 3D surface mesh model; calculating a center point at each interval by triangulating the distance between the intersection of each projected ray and the 3D surface mesh model; adjusting the direction of travel at each interval based on direction vectors between adjacent calculated center points such that ray projection at the predefined intervals occurs in at least three directions perpendicular to the adjusted direction of travel at each interval; and calculating a centerline of the isolated anatomical feature based on the calculated center points from the start point to the end point.

[0015] Generating, by the server, physiological information associated with the selected pathology for the 3D surface mesh model can include: computing a centerline for the isolated anatomical feature; determining a start point and an end point for the isolated anatomical feature and a directional vector from the start point to the end point; establishing cut planes at predefined intervals along the centerline based on the directional vector from the start point to the end point, each cut plane being perpendicular to a direction of travel of the centerline at each interval; projecting a ray in the cut plane at each interval to determine a location of an intersection on the 3D surface mesh model relative to the centerline; and computing a length across the 3D surface mesh model based on the determined location of the intersection at each interval.

[0016] Generating, by the server, physiological information associated with the selected pathology for the 3D surface mesh model can include: determining a start point and an end point for the isolated anatomical feature; taking slices at predefined intervals along an axis from the start point to the end point; computing a cross-sectional area for each slice bounded by a perimeter of the isolated anatomical feature; and generating a heat map for the isolated anatomical feature based on the cross-sectional area for each slice.

[0017] Generating, by the server, physiological information associated with the selected pathology for the 3D surface mesh model can include: determining a start point and an end point for the isolated anatomical feature; computing a centerline for the isolated anatomical feature; determining directional travel vectors between adjacent points along the centerline; computing a change magnitude of directional travel vectors between adjacent points along the centerline; and generating a heat map for the isolated anatomical feature based on the change magnitude of directional travel vectors between adjacent points along the centerline.

[0018] In some embodiments, the generated physiological information associated with the selected pathology for the 3D surface mesh model can include an associated timestamp, such that the method further includes: recording, by the server, the generated physiological information and the associated timestamp; and computing, by the server, a change over time between the recorded physiological information indicative of a progression of the selected pathology based on the associated timestamp. Thus, the method can further include: computing, by the server, a magnitude of the change over time between the recorded physiological information; and generating, by the server, a heat map for the isolated anatomical feature based on the magnitude of the change over time between the recorded physiological information.

[0019] Extracting, by the server, information from the 3D surface mesh model based on the selected pathology can include isolating an anatomical feature from the one or more classified patient-specific anatomical features based on the selected pathology, analyzing features of the isolated anatomical feature with an anatomical feature database to identify one or more landmarks of the isolated anatomical feature, associating the one or more identified landmarks with the pixels of the medical image, and generating a 3D surface mesh model that defines a surface of the isolated anatomical feature that includes the identified landmarks. Further, the method can further include identifying, by the server, a guided trajectory for performing a surgical procedure based on the selected pathology and the one or more identified landmarks from a surgical instrument database, and displaying the guided trajectory to a user.

[0020] Additionally, the method can further include receiving, by the server, patient demographic data, identifying, by the server, one or more medical devices from a medical device database based on the patient demographic data and the generated physiological information associated with the selected pathology of the 3D surface mesh model, and displaying the identified one or more medical devices to a user. Further, the method can further include receiving, by the server, patient demographic data, identifying, by the server, one or more treatment options from a surgical instrument database based on the patient demographic data and the generated physiological information associated with the selected pathology of the 3D surface mesh model, and displaying the identified one or more treatment options to a user.

[0021] Extracting, by the server, information from the 3D surface mesh model based on the selected pathology can include isolating an anatomical feature from the one or more classified patient-specific anatomical features based on the selected pathology, analyzing features of the isolated anatomical feature with an anatomical feature database to identify one or more landmarks of the isolated anatomical feature, analyzing features of the one or more landmarks with a reference fracture database to detect a fracture of the isolated anatomical feature, and generating a 3D surface mesh model of the isolated anatomical feature that includes the one or more identified landmarks and the detected fracture. Accordingly, the method can further include matching the 3D surface mesh model of the isolated anatomical feature with the reference fracture database to classify the detected fracture.

[0022] The method can further include: classifying, by the server, the one or more patient-specific anatomical features; separating, by the server, the classified one or more patient-specific anatomical features into individual anatomical features; and mapping, by the server, the individual anatomical features to original gray scale values of the medical image and removing background within the medical image, and wherein the generated 3D surface mesh model defines surfaces of the individual anatomical features, or includes volume renderings defined by mapping specific color or transparency values to the classified one or more patient-specific anatomical features. In some embodiments, the segmentation algorithm can include at least one of: a threshold-based, decision tree, chain decision forest, or neural network method. The physiological information associated with the selected pathology can include at least one of: diameter, volume, density, thickness, surface area, Hounsfield unit standard deviation, or mean value.

[0023] According to another aspect of the disclosure, a system for multi-modal analysis of patient-specific anatomical features from medical images is provided. The system can include a server and can: receive a medical image of a patient and metadata associated with the medical image indicative of a selected pathology; automatically process the medical image using a segmentation algorithm to label pixels of the medical image and generate a score indicative of a likelihood that the pixels are correctly labeled; probabilistically match associated groups of the labeled pixels to an anatomical knowledge dataset using an anatomical feature recognition algorithm to classify one or more patient-specific anatomical features within the medical image; generate a 3D surface mesh model defining surfaces of the one or more classified patient-specific anatomical features; extract information from the 3D surface mesh model based on the selected pathology; and generate physiological information associated with the selected pathology for the 3D surface mesh model based on the extracted information. For example, the information extracted from the 3D surface mesh model can include a 3D surface mesh model of an anatomical feature isolated from the one or more classified patient-specific anatomical features based on the selected pathology.

[0024] According to yet another aspect of the disclosure, there is provided a non-transitory computer-readable memory medium having instructions stored thereon that, when loaded by at least one processor, cause the at least one processor to: receive a medical image of a patient and metadata associated with the medical image indicative of a selected pathology; automatically process the medical image using a segmentation algorithm to label pixels of the medical image and generate scores indicative of a likelihood that the pixels are correctly labeled; probabilistically match associated groups of the labeled pixels to an anatomical knowledge dataset using an anatomical feature recognition algorithm to classify one or more patient-specific anatomical features within the medical image; generate a 3D surface mesh model that bounds a surface of the one or more classified patient-specific anatomical features; extract information from the 3D surface mesh model based on the selected pathology; and generate physiological information associated with the selected pathology for the 3D surface mesh model based on the extracted information. BRIEF DESCRIPTION OF DRAWINGS

[0025] FIG. 1 Some example components that can be included in a multi-modal analysis platform according to the principles of the disclosure are shown.

[0026] FIG. 2 is a flowchart illustrating exemplary method steps for multi-modal analysis of patient-specific anatomical features from medical images according to the principles of the disclosure.

[0027] FIG. 3 is a flowchart illustrating exemplary method steps for generating volume measurements of patient-specific anatomical features according to the principles of the disclosure.

[0028] FIG. 4A illustrates cross-sectional area measurements at various points along a blood vessel, and FIG. 4B illustrates determining a volume based on cross-sectional area measurements according to the principles of the disclosure.

[0029] FIG. 5 is a flowchart illustrating exemplary method steps for generating centerline measurements of patient-specific anatomical features according to the principles of the disclosure.

[0030] FIG. 6 illustrates determining a centerline according to the principles of the disclosure.

[0031] FIG. 7A illustrates a center point of a blood vessel, FIG. 7B illustrates a centerline of a blood vessel, FIG. 7C illustrates a measurement of a length of a centerline of a blood vessel, and FIG. 7D illustrates a blood vessel delineated across a medical image.

[0032] FIG. 8Aillustrating a start point and an end point of a patient-specific anatomical feature, FIG. 8B illustrating a centerline of a patient-specific anatomical feature, FIG. 8C illustrating centerlines of various patient-specific anatomical features, and FIG. 8D illustrating a centerline of a network of patient-specific anatomical features.

[0033] FIG. 9 is a flowchart illustrating exemplary method steps for generating a surface length measurement of a patient-specific anatomical feature in accordance with the principles of the present disclosure.

[0034] FIG. 10 illustrating determining a surface length in accordance with the principles of the present disclosure.

[0035] FIG. 11 illustrating a surface length of a patient-specific anatomical feature.

[0036] FIG. 12 is a flowchart illustrating exemplary method steps for generating a heat map of a patient-specific anatomical feature based on volume in accordance with the principles of the present disclosure.

[0037] FIG. 13A and 13B illustrating a volume-based heat map of a patient-specific anatomical feature.

[0038] FIG. 14 is a flowchart illustrating exemplary method steps for generating a heat map of a patient-specific anatomical feature based on tortuosity in accordance with the principles of the present disclosure.

[0039] FIG. 15 illustrating a tortuosity-based heat map of a patient-specific anatomical feature.

[0040] FIG. 16 is a flowchart illustrating exemplary method steps for generating a 3D surface mesh model of a patient-specific anatomical feature with identified landmarks in accordance with the principles of the present disclosure.

[0041] FIG. 17A illustrating exemplary method steps for mapping identified landmarks of a patient-specific anatomical feature to a 3D surface mesh model in accordance with the principles of the present disclosure.

[0042] FIG. 17B illustrating identified landmarks of a patient-specific anatomical feature mapped to a 3D surface mesh model.

[0043] FIG. 18 is a flowchart illustrating exemplary method steps for identifying medical devices and treatment options for a pathology in accordance with the principles of the present disclosure.

[0044] FIG. 19Aillustrates a pathology of a bone, and FIG. 19B and 19C illustrates various medical devices that can be used to treat a pathology.

[0045] FIG. 20 is a flowchart illustrating exemplary method steps for detecting and classifying a fracture of a patient-specific anatomical feature according to the principles of the present disclosure.

[0046] FIG. 21A to 21D illustrates mapping a detected fracture of a patient-specific anatomical feature to a 3D surface mesh model according to the principles of the present disclosure.

[0047] FIG. 22 is a flowchart illustrating exemplary method steps for tracking a progression of a pathology over time according to the principles of the present disclosure.

[0048] FIG. 23A to 23F illustrates various progressions of a pathology over time.

[0049] FIG. 24A to 24F illustrates a heat map of various progressions of a pathology over time.

[0050] FIG. 25 is a flowchart illustrating exemplary method steps for analyzing physiological parameters of an individual anatomical feature according to the principles of the present disclosure.

[0051] FIG. 26 illustrates generating a 3D volume rendering of an individual anatomical feature according to the principles of the present disclosure.

[0052] FIG. 27A illustrates an original medical image of a patient-specific anatomical feature, FIG. 27B illustrates an individual anatomical feature overlaid on the original medical image, FIG. 27C illustrates the individual anatomical feature with background removed, and FIG. 27D illustrates a 3D volume rendering of the individual anatomical feature.

[0053] FIG. 28A and 28B illustrates exemplary method steps for measuring an occlusion of a patient-specific anatomical feature according to the principles of the present disclosure.

[0054] FIG. 29 is a flowchart illustrating exemplary method steps for analyzing physiological parameters of an individual anatomical feature according to the principles of the present disclosure.

[0055] FIG. 30A to 30E illustrates generating a measurement of a patient-specific anatomical feature according to the principles of the present disclosure.

[0056] FIG. 31illustrates a weight mask generated using a Euclidean distance weighting method, and its effect on a loss function, in accordance with the principles of the present disclosure.

[0057] FIG. 32 illustrates various segmentations of bone within medical images for training purposes, in accordance with the principles of the present disclosure.

[0058] FIG. 1 illustrates various segmentations of myocardium within medical images for training purposes, in accordance with the principles of the present disclosure. DETAILED DESCRIPTION

[0059] REFERENCE FIG. 3 to 24F components that can be included in a multi-modal analysis platform 100. The platform 100 can include one or more processors 102, communication circuitry 104, a power supply 106, a user interface 108, and / or a memory 110. One or more electrical components and / or circuits can perform some or all of the functions of the various components described herein. Although described separately, it should be appreciated that the electrical components need not be separate structural elements. For example, the platform 100 and the communication circuitry 104 can be embodied in a single chip. In addition, while the platform 100 is described as having a memory 110, a memory chip can be provided separately.

[0060] The platform 100 can contain memory, and / or be coupled via one or more buses to read information from or write information to memory. The memory 110 can include processor cache memory, including multiple levels of hierarchical cache memory, with different levels having different capabilities and access speeds. The memory can also include random access memory (RAM), other volatile memory, or non-volatile memory. The memory 110 can be RAM, ROM, flash memory, other volatile memory or non-volatile memory, or other known memory, or some combination thereof, and preferably includes storage devices in which data can be selectively stored. For example, the storage devices can include hard disk drives, optical disks, flash memory, and Zip drives. Programmable instructions can be stored on the memory 110 to perform algorithms for automatically segmenting and identifying patient-specific anatomical features within medical images, including corresponding anatomical landmarks, generating 3D surface mesh models of patient-specific anatomical features, and extracting information from the 3D surface mesh models based on selected pathologies to generate physiological information for patient-specific anatomical features.

[0061] The platform 100 can incorporate a processor 102, which can consist of one or more processors and can be a general -purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any suitable combination thereof, designed to perform the functions described herein. The platform 100 can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors with a DSP core, or any other such configuration.

[0062] In conjunction with the firmware / software stored in memory, the platform 100 can execute an operating system (e.g., operating system 124), such as Windows, Mac OS, Unix, or Solaris 5.10. The platform 100 can also execute software applications stored in memory. By way of example, the software can be programmed in any suitable programming language including, for example, C++, PHP, or Java, as known to those of skill in the art.

[0063] The communication circuitry 104 can include circuitry that allows the platform 100 to communicate with image capture devices and / or other computing devices for receiving image files (e.g., 2D medical images) and metadata associated therewith indicative of patient-specific pathologies. Additionally or alternatively, image files can be uploaded directly to the platform 100. The communication circuitry 104 can be configured for wired and / or wireless communication via a network, such as the Internet, a telephone network, a Bluetooth network, and / or a WiFi network, using techniques known in the art. The communication circuitry 104 can be a communication chip as known in the art, such as a Bluetooth chip and / or a WiFi chip. The communication circuitry 104 permits the platform 100 to transmit information, such as 3D surface mesh models, physiological measurements, and treatment options, locally and / or to remote locations (e.g., a server).

[0064] The power supply 106 can supply alternating current or direct current. In direct current embodiments, the power supply can include a suitable battery, such as a replaceable or rechargeable battery, and the device can include circuitry for charging the rechargeable battery and a detachable power cord. The power supply 106 can be charged by a charger via an inductive coil within the charger. Alternatively, the power supply 106 can be a port that allows the platform 100 to be plugged into a conventional wall outlet, e.g., via a cord with an AC / DC power converter, and / or a USB port for powering the components within the platform 100.

[0065] The user interface 108 can be used to receive input from a user and / or provide output to the user. For example, the user interface 108 can include a touchscreen, a display, switches, dials, lights, etc. Thus, the user interface 108 can display information such as 3D surface mesh models, physiological measurements, heat maps, a list of medical devices that can be used for a patient-specific pathology, treatment options, etc. to facilitate diagnosis, pre-operative planning, and treatment for specific use cases, as described in further detail below. Further, the user interface 108 can receive user input including patient demographic data (e.g., patient height, age, weight, medical history, patient-specific pathology, etc.) and feedback from a user based on information displayed to the user (e.g., corrected anatomical feature identification, physiological measurements, specific anatomical feature selection) so that the platform 100 can adjust the information accordingly. In some embodiments, the user interface 108 is not present on the platform 100, but is instead provided on a remote external computing device that is communicatively connected to the platform 100 via the communication circuitry 104.

[0066] As one example of a non-transitory computer-readable medium, the memory 110 can be used to store an operating system (OS) 124, an image receiver module 112, a segmentation module 114, an anatomical feature identification module 116, a 3D surface mesh model generation module 118, an anatomical feature information extraction module 120, and a physiological information generation module 122. The modules are provided in the form of computer-executable instructions that are executable by the processor 102 to perform various operations in accordance with the present disclosure.

[0067] The image receiver module 112 can be executed by the processor 102 to receive standard medical images (e.g., 2D and / or 3D medical images) of one or more patient-specific anatomical features acquired from one or a combination of CT, MRI, PET, and / or SPECT scanners. The medical images can be formatted in a standard (e.g., DICOM) compliant manner. The medical images can include metadata embedded therein that indicates a patient-specific pathology associated with the patient-specific anatomical features in the medical images. The image receiver module 112 can pre-process the medical images for further processing and analysis, as described in further detail below. For example, the medical images can be pre-processed to generate a set of new medical images that are uniformly distributed according to a predetermined orientation based on the patient-specific anatomical features, the patient’s specific pathology, or any downstream applications (e.g., pre-operative training and / or for machine learning / neural network training purposes). Further, the image receiver module 112 can receive medical images acquired simultaneously from multiple perspectives of the patient-specific anatomical features to enhance segmentation of the patient-specific anatomical features.

[0068] Segmentation module 114 can be executed by processor 102 to automatically segment a medical image received by image receiver module 112, e.g., to assign a label to each pixel of the medical image. The assigned label can represent a particular tissue type, e.g., bone, soft tissue, blood vessels, organs, etc. In particular, segmentation module 114 can use a machine learning based image segmentation technique that includes one or a combination of the following techniques: threshold based, decision tree, chain decision forest, or neural network methods, such that the results of each technique can be combined to produce a final segmentation result, as described in U.S. Patent No. 11,138,790 and U.S. Patent Application Publication No. 2021 / 0335041 by Haslam, both assigned to the assignee of the present disclosure, and both of which are incorporated herein in their entirety. The machine learning based image segmentation technique can be trained using a knowledge database that includes pre-labeled medical images (i.e., ground truth data).

[0069] For example, segmentation module 114 can apply a first segmentation technique (e.g., threshold based segmentation) to assign a label to each pixel of a medical image based on whether a characteristic (e.g., Hounsfield value) of the pixel meets / exceeds a predetermined threshold. The predetermined threshold can be determined via, e.g., histogram analysis, as described in U.S. Patent No. 11,138,790. Further, segmentation module 114 can extend the threshold based segmentation technique by using a logic or probability function to compute a score regarding the likelihood that a pixel is of a tissue type labeled by the threshold based segmentation.

[0070] Segmentation module 114 can then apply a decision tree to each labeled pixel of the medical image to thereby classify / label each pixel based at least in part on, but not exclusively on, the score. As described in U.S. Patent No. 11,138,790, a decision tree can be applied to a subset of the labeled pixels by subsampling the medical image such that segmentation module 114 can restore a full segmentation of the medical image by upsampling the labeled pixels of the subset of pixels of the medical image using a standard interpolation method. The decision tree can consider the score as well as, for example, how many pixels that look almost like bone are near the pixel in question, how many pixels that look exactly like bone are near the pixel in question, or how strong the overall gradient of the image is at the given pixel. For example, if the pixel in question is labeled as bone with a score of 60 / 100, a first decision node of the decision tree can ask how many pixels that look almost like bone are near the pixel in question. If the answer is close to zero, meaning that there are very few pixels near the pixel in question that look almost like bone, segmentation module 114 can determine that the pixel in question is not bone despite the previous bone label having a score of 60 / 100. A new score can then be generated as to the likelihood that the pixel in question is correctly labeled by the decision tree algorithm. Thus, applying a decision tree to the pixels of the medical image can result in a more accurate final segmentation result with less noise. One of ordinary skill in the art will appreciate that the decision tree can consider other properties that can be used to determine the label of a pixel.

[0071] Additionally or alternatively, segmentation module 114 can apply a chained decision forest in which the results of the initial / previous decision tree for the same pixel in question and the results of another segmentation technique (e.g., a neural network) can be fed into a new decision tree along with the scores associated with the results. For example, the new decision tree can ask one or more questions as described above to determine whether each of the previous segmentation techniques correctly labeled the pixel in question. Thus, if the initial / previous decision tree labeled the pixel in question as bone and the neural network labeled the pixel in question as non-bone, the new decision tree can determine that the pixel in question is bone based on the responses to the one or more questions asked by the chained decision forest such that the label assigned by the neural network for the pixel in question is discarded. Furthermore, each forest node can be treated as a simple classifier that produces a score as to the likelihood that the pixel is correctly labeled by each subsequent new decision tree. Thus, applying a chained decision forest to the pixels of the medical image can result in a more accurate final segmentation result.

[0072] The anatomical feature identification module 116 can be executed by the processor 102 to identify one or more patient-specific anatomical features within the medical image by probabilistically matching the pixels labeled by the segmentation module 114 to a dataset of anatomical knowledge within a knowledge database. Specifically, as described in U.S. Patent No. 11,138,790, the anatomical feature identification module 116 can initially group the labeled pixels, for example, by establishing links between different labeled / classified pixels based on similarity between the pixels labeled by the segmentation module 114. For example, all pixels labeled as “bone” can be grouped / linked together in a first group, all pixels labeled as “organ” can be grouped / linked together in a second group, and all pixels labeled as “vessel” can be grouped / linked together in a third group.

[0073] The anatomical feature identification module 116 can then use an anatomical feature identification algorithm to explore the dataset of anatomical knowledge to identify patient-specific anatomical features within the medical image by establishing links between the grouped labeled pixels and existing knowledge within the dataset of anatomical knowledge. For example, the existing knowledge can include known information about various anatomical features (e.g., tissue types, such as bone, vessels, or organs, etc.) as well as pre-labeled ground truth data that can be used to train various segmentation algorithms, represented as nodes within a graph database of the dataset of anatomical knowledge.

[0074] The medical ontology of the existing knowledge of anatomical features within the graph database can be represented as a series of nodes grouped together by at least one of: function, proximity, anatomical grouping, or frequency of occurrence in the same medical image scan. For example, nodes representing organs can be grouped together as hearts because they are within a predetermined proximity of each other, all proximate to nodes representing vessels grouped together as aorta, and have a high frequency of occurrence in the same medical image scan. Thus, the anatomical feature identification algorithm can identify patient-specific anatomical features within the medical image by exploring the graph database to determine which group of nodes is most similar to the grouped labeled pixels, for example, based on the established links between the grouped labeled pixels and the group of nodes. Further, the anatomical feature identification module 116 can generate a score representing the likelihood that the patient-specific anatomical feature was correctly identified by the anatomical feature identification algorithm.

[0075] 3D surface mesh model generation module 118 can be executed by processor 102 to generate 3D surface mesh models of patient-specific anatomical features within the medical images based on the results of the segmentation algorithm described above and the results of the anatomical feature identification algorithm, and to extract the 3D surface mesh models from the scalar volumes to generate 3D printable models. For example, as described in U.S. Patent No. 11,138,790, the 3D surface mesh models can have the following properties: all unconnected surfaces are closed manifolds, proper supports are used to keep unconnected surfaces / volumes in place, proper supports are used to facilitate 3D printing, and / or no surface volumes are hollow, such that the 3D surface mesh models are 3D printable. Furthermore, 3D surface mesh model generation module 118 can generate 3D surface mesh models of patient-specific anatomical features that include any corresponding landmarks of the anatomical features, as described in further detail below.

[0076] Anatomical feature information extraction module 120 can be executed by processor 102 to extract information from the 3D surface mesh models generated by 3D surface mesh model generation module 118. For example, anatomical feature information extraction module 120 can extract one or more specific anatomical features from the 3D surface mesh models representing patient-specific anatomical features within the medical images based on a selected pathology indicated in the metadata received by image receiver module 112. Alternatively, platform 100 can receive information indicating a selected pathology associated with the medical images directly from a user via user interface 108, e.g., along with patient demographic data and medical history. Thus, if a given patient’s specific pathology is known, anatomical feature information extraction module 120 can automatically extract a 3D surface mesh model of the specific anatomical features including the pathology from the 3D surface mesh models generated by 3D surface mesh model generation module 118.

[0077] Physiological information generation module 122 can be executed by processor 102 to generate physiological information associated with the selected pathology for the 3D surface mesh models based on the information extracted by anatomical feature information extraction module 120. For example, based on the selected pathology, physiological information generation module 122 can perform calculations to determine physiological measurements relevant to the diagnosis and / or treatment of the pathology, e.g., by providing a list of medical devices and / or treatment options suitable for treating the pathology based on measurements of the anatomical features and patient demographic data. The list of medical devices and / or treatment options can be extracted by physiological information generation module 122 from a medical device database or a surgical instrument database. As described in further detail below with reference to FIG. 2, physiological information generation module 122 can generate a list of medical devices and / or treatment options suitable for treating the pathology based on the measurements of the anatomical features and patient demographic data. FIG. 2Further in detail, the physiological measurements associated with the selected pathology determined by the physiological information generation module 122 can include, but are not limited to, volume, cross-sectional area, diameter, centerline, surface, density, thickness, tortuosity, size and location of a fracture, blood clots, occlusions, and growth rate over time of an anatomical feature and / or corresponding landmark. In addition, the physiological information generated by the physiological information generation module 122 can be used to generate heat maps to facilitate visual observation of physiological measurements of patient-specific anatomical features.

[0078] Referring now to FIG. 3 to 24F , an exemplary method 200 for multi-modal analysis of patient-specific anatomical features from medical images using the platform 100 is provided. At step 202, a medical image and metadata associated with the medical image indicative of a selected pathology can be received by the image receiver module 112. As described above, information indicative of a selected pathology can be received directly via user input along with patient demographic data. At step 204, the segmentation module 114 can automatically process the medical image using a segmentation algorithm to label pixels of the medical image and generate scores indicative of the likelihood that the pixels are correctly labeled. For example, the segmentation algorithm can label pixels of the medical image using one or a combination of various machine learning-based image segmentation techniques trained with a knowledge dataset utilizing pre-labeled medical images.

[0079] At step 206, the anatomical feature identification module 116 can group pixels labeled at step 204 together based on similarity and probabilistically match associated groups of labeled pixels to an anatomical knowledge dataset using an anatomical feature identification algorithm to classify one or more patient-specific anatomical features within the medical image. At step 208, the 3D surface mesh model generation module 118 can generate a 3D surface mesh model defining a surface of the one or more classified patient-specific anatomical features within the medical image. At step 210, the anatomical feature information extraction module 120 can extract information from the 3D surface mesh model based on the selected pathology and the physiological information generation module 122 can generate physiological information associated with the selected pathology for the 3D surface mesh model based on the extracted information. Reference is made below to FIG. 3 The generated physiological information is described in further detail.

[0080] Referring now to FIG. 2 , an exemplary method 300 for generating volume measurements of patient-specific anatomical features is provided. As described above with respect to FIG. 4Adescribed in step 210 of method 200 for multi-modal analysis of patient-specific anatomical features from medical images, such as volume measurements of patient-specific anatomical features associated with a selected pathology, and other physiological information can be generated from the generated 3D surface mesh model. For example, at step 302, a particular anatomical feature can be isolated from the patient-specific anatomical features within the medical image for further analysis based on, for example, a selected pathology indicated by metadata associated with the medical image, such that a 3D surface mesh model of the isolated anatomical feature can be extracted and recorded from the 3D surface mesh model of the patient-specific anatomical features. Thus, the anatomical feature including only the pathology can be further analyzed to generate physiological information associated with the pathology.

[0081] At step 304, starting and ending points of the isolated anatomical feature, for example, at opposite ends of the isolated anatomical feature, are determined. For example, the starting and ending points can be determined via a machine learning algorithm that explores an anatomical knowledge dataset to derive the starting and ending points of the isolated anatomical feature. At step 306, a predetermined step size can be determined such that slices can be taken at regular intervals defined by the predetermined step size along an axis of the isolated anatomical feature. For example, the axis can be a centerline of the isolated anatomical feature determined based on a direction vector extending from the starting point to the ending point, as described in further detail below. Thus, slices of the isolated anatomical feature can be taken at each interval perpendicular to a direction of travel along the centerline in a direction from the starting point and to the ending point.

[0082] At step 308, using standard computational functions, a cross-sectional area at each slice of the isolated anatomical feature can be computed, the cross-sectional area being defined by a perimeter of the isolated anatomical feature, as shown in FIG. 4A For example, the cross-sectional area of an automatically segmented label of a particular portion of an anatomical structure, such as a mitral or aortic valve anatomical structure, can be computed based on derivatives of the maximum two cross-sections of the anatomical structure, for example, using Ax B x n. In the case of an aneurysm, this data can be used to automatically provide a neck-to-dome ratio of the anatomical structure for a surgical physician. FIG. 4B FIG. 3 illustrates three slices along an isolated anatomical feature, for example, an aorta when the associated pathology is an aneurysm, whose cross-sectional areas are computed and displayed on a 3D surface mesh model of the aorta. FIG. 3 FIG. 4 illustrates how slices can be taken along an axis of a complex structure for the purpose of computing cross-sectional areas of the slices.

[0083] Referring again to FIG. 5At step 310, the 3D volume between each adjacent slice can be extrapolated based on the cross-sectional area of ​​the isolated anatomical feature at adjacent slices, such that the total volume of the isolated anatomical structure can be determined based on the extrapolated 3D volume, for example, by obtaining the sum of the extrapolated 3D volumes. Alternatively, the automatically segmented and labeled volume of a specific portion of the isolated anatomical feature (e.g., the left atrial appendage of the heart) can be calculated based on the number of voxels within the semantically labeled portion of the anatomical structure, such that the volume can be displayed to the user for evaluation.

[0084] Now for reference FIG. 2 This provides an exemplary method 500 for generating centerline measurements of patient-specific anatomical features. (As mentioned above regarding...) FIG. 6 As described in step 210 of the method 200 for multimodal analysis of patient-specific anatomical features from medical images, physiological information such as centerline measurements of patient-specific anatomical features associated with a selected pathology can be generated based on the resulting 3D surface mesh model. For example, at step 502, specific anatomical features can be isolated from patient-specific anatomical features within the medical image based on the selected pathology, as described above, so that the 3D surface mesh model of the isolated anatomical features can be extracted and recorded from the 3D surface mesh model of the patient-specific anatomical features.

[0085] At step 504, the start and end points of the isolated anatomical feature are determined, for example, at opposite ends of the isolated anatomical feature, such that a direction vector extending from the start point toward the end point can be determined. For example, the start and end points can be determined via a machine learning algorithm that explores an anatomical knowledge dataset to derive the start and end points of the isolated anatomical feature. Furthermore, the start and end points may be close to the bounding box of the 3D surface mesh model and lie on a common plane. Additionally, an initial direction of travel consistent with the direction vector extending from the start point to the end point can be determined.

[0086] At step 506, a predetermined step size can be determined such that cutting planes can be established along the axis of the isolated anatomical feature at regular intervals defined by the predetermined step size. The cutting plane at each interval can be perpendicular to the travel direction associated with the interval. For example, the initial cutting plane can be perpendicular to an initial travel direction based on a direction vector extending from the start point to the end point. Furthermore, multiple rays (e.g., three rays) can be ray-projected radially outward toward the perimeter of the isolated anatomical feature in multiple predefined directions perpendicular to the travel direction along the cutting plane at each interval, allowing the location of the intersection points of the projected rays and the 3D surface mesh model to be determined. For example, such as... FIG. 6As shown, in the direction of travel from the starting point SP to the ending point EP, the first set of three projected rays may intersect with the 3D surface mesh model of the isolated anatomical feature (e.g., blood vessel V) at points 602a, 602b, and 602c. At step 506, if it is determined that the point from which the ray is projected is outside the 3D surface mesh model, then that point can be moved into the 3D surface mesh model.

[0087] At step 508, the center point (e.g., CP1) of the isolated anatomical feature in the cutting plane at each interval can be determined, for example, by triangulation of the distance between each of the intersection points of the isolated anatomical features (e.g., points 602a, 602b, 602c). At step 510, a new direction of travel can be determined at each interval based on a direction vector extending from the previous center point of the previous interval to the current center point. For example, in FIG. 6 In the first interval, the new direction of travel can be consistent with the direction vector extending from the starting point SP to the center point CP1. If the isolated anatomical feature is a branching vessel, then steps 506 to 510 can be repeated across both branches of the vessel to generate the centerline of each branch of the 3D surface mesh model of the vessel.

[0088] Method 500 can repeat steps 506 to 510 until the end point EP is reached. For example, as FIG. 7A As shown, at the second interval, three rays can be projected along a cutting plane perpendicular to the direction of travel defined by the direction vector extending from the starting point SP to the center point CP1. The distances between the intersection points 604a, 604b, and 604c of the projected rays, along with the 3D surface mesh model, can be triangulated to determine the center point CP2 at the second interval. The previous direction of travel can then be adjusted to a new direction defined by the direction vector extending from the center point CP1 to the center point CP2. Similarly, at the third interval, three rays can be projected along a cutting plane perpendicular to the direction of travel defined by the direction vector extending from the center point CP1 to the center point CP2. The distances between the intersection points 606a, 606b, and 606c of the projected rays, along with the 3D surface mesh model, can be triangulated to determine the center point CP3 at the third interval. The previous direction of travel can then be adjusted to a new direction defined by the direction vector extending from the center point CP2 to the center point CP3. As described above, steps 510 to 512 can be repeated until the end point EP is reached, thereby determining a series of center points CP along the axis of the isolated anatomical feature, such as... FIG. 6 As shown in the diagram. Therefore, as described above, beyond the end point EP, the point from which the ray is projected will be outside the 3D surface mesh model, making it impossible for the point to return into the 3D surface mesh model, thereby indicating the end of the centerline of the isolated anatomical feature.

[0089] At step 512, a centerline for the isolated anatomical feature can be determined based on all center points (e.g., CP1, CP2, CP3…CPn). For example, the centerline could be a line drawn through all the calculated center points of the isolated anatomical feature, such as… FIG. 7B As shown in the image. FIG. 7A The diagram illustrates the centerline CL of the isolated anatomical features as passing through... FIG. 7C The line is drawn from all the center points CP. Therefore, as FIG. 7D As shown in the figure, the total length of the centerline CL of the isolated anatomical features can be determined. FIG. 7A to 7C Diagram Explanation FIG. 8A to 8D The isolated anatomical features are captured in a 3D surface mesh model of the original medical image.

[0090] Now for reference FIG. 8A Method 500 can be used to determine the center lines of a vast network of patient-specific anatomical features. For example, FIG. 8B The diagram illustrates the starting and ending points determined for a 3D surface mesh model of isolated anatomical features. FIG. 8C The diagram illustrates the centerline CL determined by isolated anatomical features mapped onto the original medical image. FIG. 8D The diagram illustrates the centerline CL, which includes the anatomical features of multiple blood vessels, and FIG. 9 The diagram illustrates the central line CL, which shows the anatomical features of a vast network of blood vessels.

[0091] Now for reference FIG. 2 This provides an exemplary method 900 for generating surface length measurements of patient-specific anatomical features. (As mentioned above regarding...) FIG. 10 As described in step 210 of the method 200 for multimodal analysis of patient-specific anatomical features from medical images, physiological information such as surface length measurements of patient-specific anatomical features associated with a selected pathology can be generated based on the resulting 3D surface mesh model. For example, at step 902, specific anatomical features can be isolated from patient-specific anatomical features within the medical image based on the selected pathology, as described above, so that the 3D surface mesh model of the isolated anatomical features can be extracted and recorded from the 3D surface mesh model of the patient-specific anatomical features.

[0092] At step 904, the centerline of the isolated anatomical feature can be determined, for example, via method 500 described above. At step 906, the start and end points of the isolated anatomical feature can be determined, for example, at opposite ends of the isolated anatomical feature. At step 908, a predetermined step size can be determined such that cutting planes can be established along the axis of the isolated anatomical feature at regular intervals defined by the predetermined step size. FIG. 11As illustrated in the middle, a cutting plane (e.g., PI, P2) at each interval of the isolated dissected feature (e.g., vessel V) can be perpendicular to a direction of travel associated with the interval (e.g., a direction of travel of a centerline at the interval as described above), and can include a center point along a centerline CL (e.g., CPI, CP2) at the respective interval and a point along a direction vector DV extending from a start point SP to an end point EP.

[0093] At step 910, a ray (e.g., rays Rl, R2) can be projected radially outward from a respective center point (e.g., center points CPI, CP2) along each cutting plane (e.g., cutting planes PI, P2) at each interval toward the 3D surface mesh model, such that a location of an intersection between the ray and the 3D surface mesh model is recorded, e.g., intersection points Dl, D2. Step 910 can be repeated at each predefined interval to determine a series of intersection points along the surface topology of the 3D surface mesh model. At step 912, a total length of a line spanning a surface extension of the 3D surface mesh model of the isolated dissected feature can be computed based on the determined intersection points, as defined by the intersection points determined at step 910. FIG. 12 A surface line SL spanning a surface topology extension of the 3D surface mesh model of the isolated dissected feature is illustrated.

[0094] For example, with respect to cardiac image segmentation, once automatic segmentation has been completed, a 3D surface mesh model of the blood vessels surrounding the heart will be created. This 3D data can then be automatically analyzed to assess specific lengths related to cardiac landmarks, which can include, but are not limited to: atrium; ventricle; aorta; vena cava; mitral valve; pulmonary valve; aortic valve; tricuspid valve; myocardium; coronary artery; left atrial appendage.

[0095] Referring now to FIG. 13A An exemplary method 1200 for generating a heat map of a patient-specific anatomical feature based on volume is provided. As described above, a cross-sectional area of an isolated dissected feature at a predefined interval along an axis of the isolated dissected feature can be determined, such that a heat map of the 3D surface mesh model can be generated based on the cross-sectional areas of the 3D surface mesh model along the axis of the isolated dissected feature. For example, at step 1202, a specific anatomical feature can be isolated from a patient-specific anatomical feature within a medical image based on a selected pathology, as described above, such that a 3D surface mesh model of the isolated dissected feature can be extracted and recorded from the 3D surface mesh model of the patient-specific anatomical feature. At step 1204, a start point and an end point of the isolated dissected feature can be determined, and an initial direction of travel consistent with a direction vector extending from the start point to the end point can be determined. At step 1206, a centerline of the isolated dissected feature can be determined, e.g., via the method 500 described above.

[0096] At step 1208, a predetermined step size can be determined such that slices can be taken at regular intervals defined by the predetermined step size along the centerline of the isolated anatomical feature. Thus, slices of the isolated anatomical feature can be taken at each interval perpendicular to the direction of travel along the centerline. At step 1210, using a standard calculation function, the cross-sectional area at each slice of the isolated anatomical feature can be calculated as defined by the perimeter of the isolated anatomical feature. At step 1210, a heat map can be generated based on the cross-sectional area at each slice of the 3D surface mesh model, thereby visually indicating the change throughout the volume of the isolated anatomical feature, as shown in FIG. 14 and 13B .

[0097] Referring now to FIG. 15 , an exemplary method 1400 for generating a heat map of a patient-specific anatomical feature based on tortuosity is provided. As described above, a direction of travel at a predefined interval of a centerline of an isolated anatomical feature can be determined such that a heat map of a 3D surface mesh model can be generated based on a magnitude of change in the direction of travel along an axis of the isolated anatomical feature. For example, at step 1402, a specific anatomical feature can be isolated from a patient-specific anatomical feature within a medical image based on a selected pathology, as described above, such that a 3D surface mesh model of the isolated anatomical feature can be extracted and recorded from a 3D surface mesh model of the patient-specific anatomical feature. At step 1404, a starting point and an ending point of the isolated anatomical feature are determined, and an initial direction of travel consistent with a directional vector extending from the starting point to the ending point can be determined. At step 1406, a centerline of the isolated anatomical feature can be determined, for example, via the method 500 described above.

[0098] At step 1408, a direction of travel at a predefined interval of the centerline of the isolated anatomical feature can be determined, for example, based on a directional vector extending between adjacent center points along the centerline, as described above. At step 1410, a magnitude of change between the directions of travel of adjacent intervals can be determined. For example, the change magnitude can be calculated using directional vectors associated with the respective directions of travel at each interval. At step 1412, a heat map can be generated based on the magnitude of change between the directions of travel of adjacent intervals along the axis of the 3D surface mesh model, thereby visually indicating the tortuosity of the isolated anatomical feature, as shown in FIG. 16 . Thus, the change magnitude (e.g., angular change) from the analysis output can be cross-referenced with an existing knowledge database of known classification angular deviations and displayed to the user. The tortuosity value can be depicted as a total change in vessel angle and scored, for example, 760 degree rotation score.

[0099] Referring now to FIG. 2This provides an exemplary method 1600 for generating 3D surface mesh models with patient-specific anatomical features that have been identified. (See above regarding...) FIG. 17A As described, FIG. 17A The medical images shown in 1702 can be automatically processed to identify, for example... FIG. 17A The patient-specific anatomical features shown in 1704 enable the generation of a 3D surface mesh model of the classified patient-specific anatomical features within the medical image. Method 1600 further identifies corresponding markers for the patient-specific anatomical features (e.g., bone notches or heart valves), enabling the depiction of these markers in the 3D surface mesh model. For example, prior to generating the 3D surface mesh model based on the classified patient-specific anatomical features, at step 1602, information indicating the specific anatomical features can be isolated from data representing the patient-specific anatomical features within the medical image based on the selected pathology, such as... FIG. 17A As shown in 1706 (anatomical drawing).

[0100] At step 1604, the features of the isolated anatomical features can be analyzed using the anatomical feature dataset to identify one or more markers of the isolated anatomical features associated with the selected pathology. For example, the anatomical feature dataset may contain knowledge of anatomical markers associated with various patient-specific anatomical features (e.g., an existing semantically labeled anatomical feature dataset), allowing the markers to be identified and individually labeled by establishing a link between the classified isolated anatomical features and the anatomical feature dataset. At step 1606, the identified labeled markers can be associated with pixels of the original medical image, such as... FIG. 17A As shown in 1708. At step 1608, a 3D surface mesh model depicting isolated anatomical features of the identified markers can be generated, the identified markers being mapped to pixels in the medical image associated with the identified markers, such as... FIG. 17B As shown in 1710.

[0101] Identified anatomical landmarks are significant points within a patient's anatomical structure that are important for their form or function, such as orientation and insertion points for other anatomical features. Identified landmarks help surgeons ensure that the landmarks correspond to specific parts of the anatomical structure and that they function and are oriented appropriately. Furthermore, identified landmarks can be used in clinical practice as markers on anatomical structures to facilitate patient diagnosis and / or treatment, for example, as initial references for anatomical guide fixation and trajectory planning. For instance, specific anatomical landmarks for each bone can be automatically detected, enabling the generation of guides for cutting and drilling the bone. Therefore, identified anatomical landmarks can serve as input for clinical function, which has significant benefits. For example, FIG. 18The following identified landmarks mapped to isolated anatomical features (e.g., scapula for shoulder replacement) are illustrated: (A) fossa center, (B) triangular region, (C) inferior angle, (D) scapular spine center. Thus, the identified landmarks can be used as a reference to guide cutting planes and drill trajectories within the bone, as well as the fixation of devices in the bone.

[0102] Referring now to FIG. 19A , an exemplary method 1800 for identifying medical devices and treatment options for a pathology is provided. For example, at step 1802, a particular anatomical feature can be isolated from a patient-specific anatomical feature within a medical image based on a selected pathology, as described above, such that a 3D surface mesh model of the isolated anatomical feature can be extracted and recorded from a 3D surface mesh model of the patient-specific anatomical feature, as shown in FIG. 19B . At step 1804, physiological parameters of the isolated anatomical feature can be analyzed, as described above, for example, to determine measurements such as volume, centerline, surface length, cross-sectional area, diameter, density, etc.

[0103] Based on the physiological parameters of the isolated anatomical feature and patient demographic data associated with the medical image, at step 1806, one or more medical devices and / or treatment options can be identified from a medical device database having knowledge of various medical devices, including their functionality and specifications, and / or a surgical instrument database having knowledge of pathology-specific treatment options. For example, the physiological parameters of the isolated anatomical feature can indicate a size of the selected pathology, such that a medical device known for treating a particular size of the selected pathology can be identified for use in treating the pathology. Further, the identified medical device can be selected from an internal inventory, for example, medical devices available or provided by a particular hospital. The knowledge datasets described herein can further include knowledge of combinations of anatomical structures and non-organic materials (e.g., polymers, metals, and ceramics), such that non-organic materials can also be automatically segmented. Additionally, the knowledge datasets can include knowledge of medical devices, which can be used as input for creating patient-specific guidelines, for example, knowledge of pre-existing implants for correcting bone pathologies. For example, known dimensions and variability of devices can be used as input in the automatic design of devices. At step 1808, the identified medical devices and / or treatment options can be displayed to a user, such that the user can make informed decisions regarding preoperative planning and treatment, as shown in FIG. 20 and 19C .

[0104] The ability to provide automatic segmentation opens up several beneficial pathology-specific applications. For example, some specific pathologies / treatments that require higher volume 3D models (virtual or physical) are listed in Table 1 below.

[0105] Table 1

[0106]

[0107]

[0108] O - Orthopedics

[0109] C - Cardiac / Heart Disease N - Neurology

[0110] G - General Practice

[0111] M - Maxillofacial

[0112] On - Oncology

[0113] IR - Interventional Radiology

[0114] Referring now to FIG. 2 , an exemplary method 2000 is provided for detecting and classifying fractures of patient-specific anatomical features. As described above with respect to FIG. 21A , a medical image (e.g., as shown in FIG. 21B ) can be automatically processed to identify patient-specific anatomical features such that a 3D surface mesh model of the classified patient-specific anatomical features within the medical image can be generated. The method 1600 further detects / identifies corresponding fractures of the patient-specific anatomical features (e.g., in bones such as tibia, fibula, or medial malleolus) such that the fractures can be delineated in the 3D surface mesh model. For example, prior to generating the 3D surface mesh model based on the classified patient-specific anatomical features, at step 2002, information indicative of a particular anatomical feature can be isolated from data representing the patient-specific anatomical features within the medical image based on a selected pathology, as shown in FIG. 21D and 21C .

[0115] At step 2004, the features of the isolated anatomical feature can be analyzed with the anatomical feature dataset to identify one or more landmarks of the isolated anatomical feature (e.g., a bony notch) that are associated with the selected pathology. As described above, the anatomical feature dataset can include knowledge of anatomical landmarks associated with various patient-specific anatomical features, such that the landmarks can be identified and individually labeled by establishing a link between the classified isolated anatomical feature and the anatomical feature dataset. At step 2006, the features of the identified landmarks can be analyzed with the reference fracture database to identify one or more fractures of the identified landmarks of the isolated anatomical feature that are associated with the selected pathology. The reference fracture database can include knowledge of various fractures associated with various patient-specific anatomical features (e.g., an existing semantically labeled reference fracture dataset), such that the fractures can be identified and individually labeled by establishing a link between the classified isolated anatomical feature and the anatomical feature dataset. At step 2008, a 3D surface mesh model of the isolated anatomical feature can be generated that depicts the identified landmarks and detected fractures F, as shown in FIG. 20B. Furthermore, at step 2010, the 3D surface mesh model can be matched with the reference fracture database to classify the fracture types. FIG. 22

[0116] Referring now to FIG. 22, FIG. 23A to 23F an exemplary method 2200 for tracking the progression of a pathology over time is provided. For example, at step 2202, a particular anatomical feature can be isolated from a patient-specific anatomical feature within a medical image based on a selected pathology, as described above, such that a 3D surface mesh model of the isolated anatomical feature can be extracted and recorded from a 3D surface mesh model of the patient-specific anatomical feature. At step 2204, physiological parameters of the isolated anatomical feature can be analyzed, as described above, for example to determine measurements such as volume, centerline, surface length, cross-sectional area, diameter, density, etc.

[0117] For example, once the automatic segmentation has been completed, 3D surface mesh models of the aneurysm and vascular anatomy can be generated. This 3D data can then be automatically analyzed to assess specific lengths related to aneurysm morphology, which can include, but are not limited to, measurements of the aneurysm neck, diameter measurements of the aneurysm at the maximum distance, and measurements of the center points of the upper and lower aneurysm necks.

[0118] At step 2206, the analyzed physiological parameters of the isolated anatomical feature can be timestamped and recorded, such that over time, there is a chronological record of the physiological parameters of a particular patient. At step 2208, changes over time between the recorded / timestamped physiological parameters can be calculated to indicate, for example, the progression and prognosis of the selected pathology. For example, FIG. 24A to 24F ​This diagram illustrates the growth of various aneurysms over time, leading to eventual rupture. Optionally, at step 2210, a thermogram can be generated to visually depict changes over time between recorded / time-stamped physiological parameters, such as... FIG. 25 As shown in the image.

[0119] Now for reference FIG. 26 , providing an exemplary method 2500 for semantic volume rendering. FIG. 26 The image 2602 is shown as a single medical image within a stack of medical images 2604. Volumetric rendering is a crucial solution adopted by medical professionals worldwide to visualize medical imaging datasets in 3D space. It works by mapping pixel characteristics, such as specific color, intensity, or opacity, to specific voxels within a 3D scene. A drawback associated with this imaging method is that overlapping and deep structures are not easily visualized in detail. Therefore, to address these drawbacks, method 2500 generates 3D surface mesh models of individual anatomical features, enabling the analysis of physiological parameters of individual anatomical features.

[0120] For example, the result of automatic image segmentation can be presented as a series of binary pixel arrays contained in a medical image, such as a DICOM file. When assembled into a volume, the binary pixel array can be used to mask regions of the source pixel volume that are irrelevant to the identified anatomical structure. The remaining Hausfield value volume can then be rendered using a standard volume rendering technique with a color transfer function, allowing pixel intensity to be determined based on the Hausfield value. Furthermore, the length of anatomical features (e.g., blood vessels) can be calculated based on the output from the automatic segmentation algorithm and subsequent 3D reconstruction. The data extracted from the 3D reconstruction can then be automatically analyzed to output the length from one specific anatomical landmark or anomaly to another, for example, the length from the aortic arch to the thrombus in the case of a stroke. In the case of blood vessels, measurements can be calculated by creating a center point on the cross-section of the blood vessel and an extrapolated center point through the blood vessel, and connecting these center points to create a centerline of the anatomical structure. This centerline can then be automatically measured and output as a length value to the user.

[0121] For example, at step 2502, the classified patient-specific anatomical features generated using the segmentation algorithm described above are depicted as binary labels, such as bone / non-bone, blood vessel / non-blood vessel, organ / non-organ, etc. At step 2504, the binary labels are separated into individual anatomical features, such as the myocardium, aorta, coronary arteries, etc. of the heart. At step 2506, the individual anatomical features are mapped to the original medical image so that only the original grayscale values ​​or Hausfield units of the individual anatomical features are displayed in the medical image, such as... FIG. 27B and FIG. 26 As shown in 2606 and 2608, background can be removed from medical images, such as FIG. 27C andFIG. 26 As shown in 2610 and 2612, only individual anatomical features depicted in their original grayscale values ​​or Hausfield units remain visible.

[0122] At step 2508, a 3D surface mesh model of each individual anatomical feature can be generated. The 3D surface mesh model defines the surface of each individual anatomical feature, such as... FIG. 26 As shown in 2614. Alternatively, specific colors of transparency values ​​can be mapped to a labeled 3D surface mesh model to produce volumetric rendering, such as... FIG. 27D and FIG. 27D As illustrated in 2616. For example, a color map of pixel intensity can be directly mapped to the intensity of 3D voxels only within a segment, allowing for the visualization of specific volumes of isolated anatomical features. Voxels can be automatically assigned specific colors depending on the intensity of the original image, which can indicate normal or absent blood flow. The ability to color specific areas of interest (such as blood clots, lacerations, or anatomical structures) allows for a deeper understanding of the specific pathology of the area.

[0123] like FIG. 28A As shown, 3D volumetric rendering can indicate the presence of blood clots / occlusions. This data can then be presented on an end-user application, allowing the 3D volumetric rendering to be rotated or otherwise manipulated and viewed. This data can also be used to indicate to the user, for example, the presence of calcification in a set of high-intensity pixels, and can be further provided by indicating the percentage of blood clots or occlusions representing calcified structures, thus providing a calcification "score." For example, predictions of occlusion / calcification can be made and applied as a mask to the original medical image, allowing the background portion of the medical image to be removed, such as... FIG. 28A As shown in 2802. Therefore, 3D surface mesh models that take into account the pixel intensity of various materials can be generated, such as FIG. 28B As shown in 2804, 2806, and 2808. FIG. 29 As shown in the figure, the size of the occlusion O depicted in a 3D volumetrically rendered blood vessel V can be measured, for example, to assist in the diagnosis and treatment of stroke patients.

[0124] 3D volumetric rendering can be user-defined or automatically exported to visualize specific features by referencing anatomical features depicted in the volumetric rendering (e.g., vascular structures, coronary arteries, blood clots within neurovascular structures), thereby indicating potential stroke. Therefore, medical images can be automatically segmented and reconstructed, for example, using a patient's CTA / XA / NM vascular imaging, to create 3D representations of both blood vessels and associated occlusions using machine learning based on semantically labeled 3D anatomical knowledge datasets readily available on mobile devices or similar platforms.

[0125] Once a 3D surface mesh model has been produced from the automatic segmentation, it is possible to make a number of measurements in the medical scan about the anatomy or pathology. In addition, the scaling information along with the reference points permit the placement of patient specific anatomical features within a physical scene. At the simplest level, physical measurements can be made of the mesh or any sub-mesh or region otherwise delineated in the physical scene, which can include: length, width, height, angle, curvature, tortuosity of the mesh, etc. In the case of filled structures, volume, surface area, and diameter measurements can also be made.

[0126] Derived properties of the material to be segmented can also be measured. At a basic level, these properties can include the thickness of the material (vessel or bone), as well as deviations from the known for normal (patient or population), which can permit the production of predictions about, for example, the likely pressure required to disrupt the material, or simply provide a visualization of the thickness and stress lines. Visualization of any of the measurements mentioned above would provide great value, as any more information available to the surgeon would aid in determining the best course of action for treatment, and would provide the ability to give an accurate analysis of the diagnosis. This can be achieved by simply overlaying the derived variables on the mesh or by providing data for additional analysis of the input / desired properties.

[0127] In addition to determining the structure of patient specific anatomical features as described above, the extracted polygonal model can further provide a convenient basis for determining a number of useful measurements that would otherwise be difficult to judge from the volumetric pixel data, such as bone and vessel size, angle and tortuosity differences, and relative proportions, density, etc. Determining these measurements would typically require careful manual evaluation of the mesh in order to identify regions of interest and meaningful reference points. However, the exploratory geometry algorithm described herein provides a reliable automated alternative. For example, the following pseudo code outlines how vessel length, diameter, and curvature information can be automatically collected without human intervention:

[0128] getVesselInfo(mesh){

[0129] - get bounding box of input mesh

[0130] - get min and max coordinates along each axis

[0131] - any vertices that exist at these extreme points can be considered to form part of the circular opening of the vessel

[0132] - construct circular / elliptical entry point by clustering previously identified extreme vertices

[0133] - get center point of vessel opening

[0134] - determine inward direction of vessel from volume

[0135] - For each entry point center

[0136] - When the projected ray does not collide with the plane defined by the vessel entry point

[0137] - Create a new measurement line

[0138] - Ray projection at different equidistant angles

[0139] - Take the farthest distance

[0140] - Proceed along the distance line

[0141] - Determine the center of the vessel diameter by calculating the center of the smallest diameter line

[0142] - (Save diameter value for later determination of thickness difference)

[0143] - Add new position to measurement line

[0144] - In case of multiple peaks in distance array

[0145] - Proceed for each branch

[0146] - Remove exit point from entry point list

[0147] - Return resulting directional path

[0148] Referring now to FIG. 30A to 30E , an exemplary method 2900 for analyzing physiological parameters of individual anatomical features is provided. Some of the steps of method 2900 can be further detailed by reference to FIG. 30A , which depicts a 2D instance of a cross-section of a vessel having a branching path. FIG. 30B A branching vessel V is illustrated. At step 2902, planes PI, P2, P3 can be constructed at the entry points of vessel V, as shown in FIG. 30C , which are bounded by the volume boundary of vessel V. At step 2904, center points CI, C2, C3 of entry planes PI, P2, P3, respectively, can be calculated, as shown in FIG. 30C . As shown in FIG. 30C , a plurality of rays can be projected from center point C3 into the structure of vessel V to determine the longest unobstructed path within vessel V. As vessel V has a branching path, two peak points PP1, PP2 are depicted in FIG. 30D . This can be determined by evaluating the number of inflection points in the distance value map. In the event that it has been determined in the algorithm that there are many forward paths at this point, each branch can be evaluated individually by branching the control flow.

[0149] At step 2906, the entire structure of the blood vessel V is traversed until a ray cast at each point along the lines Ll, L2 intersects the entry planes P2 and P3, respectively, as shown in FIG. 30E The result is a series of vertices, as shown in FIG. 31 The result is a series of vertices, as shown in

[0150] Further, in accordance with the pseudo code described above, the presence of a pathology such as an aneurism will cause the search point to fall into a loop. Whenever the point of the measuring line starts to change direction repeatedly, the algorithm can break out of the search loop and assume that an aneurism has been entered. Thus, physiological measurements of an aneurism can be determined, for example, by the following steps: determining a point around the entrance of the aneurism; constructing an entrance plane of the aneurism; determining a center point of the entrance plane; and casting a ray into the aneurism structure to determine the farthest point; and when the maximum distance has been determined, constructing a line between the entrance plane and the maximum distance point, and starting to check the perpendicular distance by ray casting.

[0151] Segmentation results can be quantified, for example, by measuring the density of the segmented region, identifying proximity to other anatomical segments, and identifying and drawing boundaries, especially with respect to oncology. Once a region is identified and drawn within a physical scene, statements can be made about the region relative to other structures within the scene. For example, drawing a tumor boundary and understanding its distance to critical structures in the anatomical neighborhood is useful to an oncologist. Further, the density of a given structure will provide clinically relevant information, for example, in the case of oncology, it will provide an understanding of hypoxia within the tumor, and in the case of a blood clot, it will allow an understanding of how the blood clot can be treated.

[0152] The ability to measure the density and thickness of anatomical regions will allow the ability to provide guidance to, for example, screw selection in trauma applications or catheter diameter in vascular applications. Further, the ability to measure the diameter along anatomical features will allow diameter measurements to be cross-referenced with a medical device database to indicate to a surgical physician the best sized device for the patient.

[0153] The machine learning based algorithms described herein can be trained and predicted on the axial axis, which is typically the axis on which the medical scan is performed. A modification to the machine learning based algorithm can involve changing the prediction function, and another modification can involve changing the training and prediction functions. For example, a modification to the machine learning based algorithm can include making predictions on all three axes, and then merging the results. This approach works best in cases where the voxels are isotropic, as in the case of the rimasys data. The merging of the predictions can follow several different strategies, for example, taking the average (mean) of the three results for a given pixel / voxel, or more complex solutions, for example, taking the axial slice plus a weighted average of the other slices. Alternatively, it can be possible to switch to a different principal axis, for example, from the axial to the sagittal.

[0154] Training the algorithm on all three axes can take advantage of additional information from different axes. Thus, an axial inference model, a sagittal inference model, and a coronal model can be trained. As described above, the results of all three predictions can be combined with a simple merging strategy. Preferably, however, the output layers of the three models can be combined in a larger network, or a collective model can be created that combines the results of them.

[0155] As described in U.S. Patent Application Publication No. 2021 / 0335041, the algorithm can work locally in 3D, which can be very expensive from a memory allocation perspective. Another approach to mitigate this limitation is to consider one cube at a time instead of one slice. The advantage of this approach is that a more relevant and direct context can be considered in the training, such that instead of considering large thick slices, the algorithm is trained on small volume cubes that slide over the entire volume.

[0156] The sandwich method described in U.S. Patent Application Publication No. 2021 / 0335041 can be extended to incorporate a larger number of slices, and can also incorporate pixels from surrounding slices in the model more explicitly. For example, multiple channels (e.g., three channels) of most image formats can be utilized to implement this compression, instead of using additional channels in the image. The number of surrounding images in a scan can typically be increased by turning surrounding images into full images. The number of surrounding images in a scan can also be increased as the size of the GPU increases.

[0157] The algorithm can implement a D-Unet version that takes into account 3D context information (via 3D convolutional kernels), and can increase the amount of slices analyzed at once by the model to provide the algorithm with more spatial context. This architecture is upgraded with improvements to the loss function, and access to more data makes the segmentation model better and better.

[0158] Further, the methods described herein can further utilize a Euclidean distance weighting method to influence loss components in the machine learning model training process. This method helps guide the learning process to focus on more important regions. For example, in orthopedic segmentation, the most difficult to detect / discover and fix errors are small connections between bones that are very close to each other; however, small holes on the inside of the bone are easier to correct. FIG. 32 FIG. illustrates a weight mask generated using the Euclidean distance weighting method, and its effect on a loss function (e.g., categorical cross-entropy).

[0159] A multi-modal approach to the ground truth data set for training is provided. In particular, there are many different segmentation label modes that can be used to adapt the training labels depending on the goal of the model to be trained. For example, since it can be difficult to define the interior material of a fractured bone, it is often segmented as hollow, and thus predictions from a fractured model trained on hollow bone labels are easier to process, as shown in Table 2 below.

[0160] Table 2: Bone segmentation label modes

[0161]

[0162]

[0163] Original Label FIG. illustrates various segmentations of bone within a medical image that use a multi-modal approach to the ground truth data for training purposes, as described above. Similarly, Table 3 illustrates heart segmentation label modes used with the multi-modal approach to the ground truth data.

[0164] Table 3: Heart segmentation label modes

[0165] Meaning Heart Heart only Context 0 0 (Context) 0 (Context) External 1 0 (Context) 0 (Context) Blood flow 2 1 (Blood flow) 1 (Blood flow) Myocardium 3 2 (Myocardium) 2 (Myocardium) Artifact 4 3 (Artifact) 0 (Context) Calcification 5 4 (Calcification) 1 (Blood flow) FIG. 33

[0166] ​ FIG. illustrates various segmentations of myocardium within a medical image that is used for ground truth data for training purposes.

[0167] These same techniques for adapting labeling patterns can be used to delineate normal tissue from pathological tissue, or lack thereof (in some instances), which would allow for semantic segmentation of pathology as the region of interest, and further allow for automatic initiation of pathology-specific workflows. Additionally, a multi-modal approach using multiple labels to distinguish between anatomical structures and pathology can be used to semantically label every anatomical feature of the human body. Examples of various modal labels can include, but are not limited to: nose; lacrimal gland; inferior turbinate; maxilla; zygoma; temporal; palatine; parietal; malleus; incus; stapes; frontal; ethmoid; vomer; sphenoid; mandible; occipital; rib 1; rib 2; rib 3; rib 4; rib 5; rib 6; rib 7; rib 8 (false); rib 9 (false); rib 10 (false); rib 11 (floating); rib 12 (floating); hyoid; sternum; cervical vertebra 1 (atlas); C2 (axis); C3; C4; C5; C6; C7; thoracic vertebra 1; T2; T3; T4; T5; T6; T7; T8; T9; T10; T11; T12; lumbar vertebra 1; L2; L3; L4; L5; sacrum; coccyx; scapula; clavicle; humerus; radius; ulna; scaphoid; lunate; triquetrum; pisiform; hamate; capitate; trapezium; trapezoid; hamulus; head of metacarpal; small multangular; large multangular; metacarpal 1; proximal phalanx 1; distal phalanx 1; metacarpal 2; proximal phalanx 2; middle phalanx 2; distal phalanx 2; metacarpal 3; proximal phalanx 3; middle phalanx 3; distal phalanx 3; metacarpal 4; proximal phalanx 4; middle phalanx 4; distal phalanx 4; metacarpal 5; proximal phalanx 5; middle phalanx 5; distal phalanx 5; hip (ilium, ischium, pubis); femur; patella; tibia; fibula; talus; calcaneus; navicular; medial cuneiform; middle cuneiform; lateral cuneiform; cuboid; metatarsal 1; proximal phalanx 1; distal phalanx 1; metatarsal 2; proximal phalanx 2; middle phalanx 2; distal phalanx 2; metatarsal 3; proximal phalanx 3; middle phalanx 3; distal phalanx 3; metatarsal 4; proximal phalanx 4; middle phalanx 4; distal phalanx 4; metatarsal 5; proximal phalanx 5; middle phalanx 5; distal phalanx 5; Willis Circle; Anterior Cerebral Artery; Middle Cerebral Artery; Posterior Cerebral Artery; Striate Artery; Brachiocephalic Artery; Right Common Carotid Artery; Right Subclavian Artery; Vertebral Artery; Basilar Artery; Posterior Cerebral Artery; Posterior Cerebral Artery; Posterior Communicating Artery; Left Common Carotid Artery; Internal Carotid Artery (ICA); External Carotid Artery (ECA); Left Subclavian Artery; Right Subclavian Artery; Internal Thoracic Artery; Thyrocervical Trunk; Costocervical Trunk; Left Subclavian Artery; Aorta; Vena Cava; Axillary; Axillary Artery; Brachial Artery; Radial Artery; Ulnar Artery; Descending Aorta; Thoracic Aorta; Abdominal Aorta; Inferior Abdominal Artery; External Iliac Artery; Femoral Artery; Popliteal Artery; Anterior Tibial Artery; Dorsal Pedal Artery; Posterior Tibial Artery; Tricuspid Valve; Pulmonary Valve; Mitral Valve; Aortic Valve; Right Ventricle; Left Ventricle; Right Atrium; Left Atrium; Liver; Kidney; Spleen; Intestine; Prostate; Cerebrum; Brainstem; Cerebellum; Pons; Medulla; Spinal Cord; Frontal Lobe; Parietal Lobe; Occipital Lobe; Temporal Lobe; Right Coronary Artery; Left Main Coronary Artery; Left Anterior Descending Branch; Left Circumflex Artery.

[0168] Hybrid data labeling for reinforcement learning is provided. For most machine learning models, creating a large data corpus for training is essential. With respect to segmentation algorithms for labeling DICOMS, as described herein, the ability to create large amounts of data for robust algorithms is limited by the resources of a skilled engineer or imaging specialist. By leveraging the initial results of a segmentation algorithm, the methods described herein can speed up the time it takes to create a large dataset. For example:

[0169] Time to segment a single image (without automation) = 10 seconds;

[0170] Assume 100,000 labeled images for robust algorithm;

[0171] Sequentially segmenting 100,000 images would take approximately 278 hours of time;

[0172] In a theoretical work example, where the model is trained four times, and the algorithm training is linear:

[0173] 0-25,000 - approximately 69 hours - training;

[0174] 25,001-50,000 (algorithm 25% complete) 52 hours - retraining;

[0175] 50,001-75,000 (algorithm 50% complete) 35 hours - retraining;

[0176] 75,001-100,000 (algorithm 75% complete) 17 hours;

[0177] Segment 100,000 images using hybrid algorithm and skilled personnel - 173 hours

[0178] The above simplified example indicates that with the help of retraining, the segmentation algorithm will be able to achieve the desired level of automation faster. Additionally, this can be furthered by retraining the algorithm after each dataset is added to the training set. This can be achieved by using cloud infrastructure and event-driven serverless computing platforms such as AWS Lambdas. Showing the user a set of updated labels after each save can greatly reduce the time to create large amounts of data.

[0179] Furthermore, most medical image segmentation applications require very high levels of accuracy, and as such, medical images can be used at their original full resolution. However, in situations where there is an inherent need to observe the entire or most of a 3D scan to detect a pathology (e.g., an aneurysm), most 2D-based methods are insufficient. Moreover, due to current hardware limitations or high cost, 3D methods can not be applicable to full resolution scans.

[0180] Thus, the methods described herein can downsample the exam volume by segmenting the vasculature in a CT scan (e.g., a neuro CT scan) using a D-Unet based architecture to find key features. This architecture looks at a small stack of 2D images, e.g., 4 slices below and 4 slices above, providing some small 3D context information. In the case of aneurysm detection, current methods can not be sufficient to distinguish aneurysms from healthy vessels, as they only look at a few 2D images at a time, which can not be enough to get the context needed to correctly identify aneurysms. This is mainly because the texture and overall appearance of an aneurysm is indistinguishable from other vasculature when viewed in isolation, e.g., in a few 2D images.

[0181] The ability to automatically identify and potentially locate and measure aneurysms, clots, and occlusions can revolutionize neurosurgery and save lives. For example, the methods described herein can use more advanced methods that can look at the entire scan from a 3D perspective in order to distinguish these abnormalities from the rest of the vasculature. Thus, the methods described herein can implement a two-step approach, where the first step uses a full resolution method to identify the vasculature in the image stack, and then in the second step, a separate model will look at a low resolution version of the scan in three dimensions. After obtaining the region of the aneurysm in the low resolution volume, the region can be co-registered with the high resolution version so that the aneurysm can be segmented from the general vasculature segmentation. This approach has great potential for other high resolution 3D volume applications where similar textured elements need to be distinguished, which require much greater context to be correctly identified.

[0182] Image preparation using real medical images to produce a model (physical or virtual) requires a certain amount of pre-screening and refinement to produce an accurate model. Thus, in order to significantly improve the quality of the final model, many transformations must be performed on the images.

[0183] For example, interpolation of images can be very easy since large datasets of existing images are available to train the algorithm. Such problems are particularly suitable for adversarial networks. Moreover, registration of images can be important since the number of cases involving multiple scanning modalities is increasing and this can require registration of CT->MRI images. For example, images from multiple scanning modalities can be registered by aligning two different datasets together, for example, if medical scans of a patient's head are provided and it is desired to get a tumor from an MRI scan and to get bones from a CT scan, then landmarks that are visible in both the MRI and CT scans can be picked in order to register the pixels and voxels at the same location. MRI scans that acquire images in multiple views / planes even in a single session can also require registration since the differences between the planes can produce significantly different views of the patient highlighting completely different aspects of the anatomy.

[0184] Rather than focusing on the individual processes themselves, the systems and methods described herein focus on how to integrate the upstream and downstream data of the platform, with particular attention to the integration required to make end-to-end possible.

[0185] This area can include all downstream integration, such as electronic medical / health records. Moreover, it is possible to collate information from EMRs (possibly associated with later outcomes, see prognosis section) which will also include any upstream integration, such as integration with courier or print bureau. The key to the value of this area is the concept of data provenance and showing the digital thread from data input through to model production of manufactured objects / virtual objects and others.

[0186] While various illustrative embodiments of the application have been described above, it will be apparent to one skilled in the art that various changes and modifications can be made therein without departing from the application. It is intended in the appended claims to cover all such changes and modifications that fall within the scope of the application.

Claims

1. A method for multi-modal analysis of patient-specific anatomical features from medical images, the method comprising: receiving, by a server, a medical image of a patient and metadata associated with the medical image indicative of a selected pathology; automatically processing, by the server, the medical image using a segmentation algorithm to label pixels of the medical image and generate scores indicative of a likelihood that the pixels are correctly labeled; probabilistically matching, by the server, associated groups of the labeled pixels to an anatomical knowledge dataset using an anatomical feature recognition algorithm to classify one or more patient-specific anatomical features within the medical image; generating, by the server, a 3D surface mesh model that bounds surfaces of the one or more classified patient-specific anatomical features; extracting, by the server, from the 3D surface mesh model, an isolated 3D surface mesh model of a patient-specific anatomical feature comprising the selected pathology based on the metadata; and generating, by the server, physiological information of the isolated 3D surface mesh model associated with the selected pathology.

2. The method of claim 1, wherein the isolated 3D surface mesh model comprises a 3D surface mesh model of an anatomical feature isolated from the one or more classified patient-specific anatomical features based on the selected pathology.

3. The method of claim 1, wherein generating, by the server, physiological information of the isolated 3D surface mesh model associated with the selected pathology comprises: determining a start point and an end point of the isolated anatomical feature; taking slices at predefined intervals along an axis from the start point to the end point; computing a cross-sectional area bounded by a perimeter of the isolated anatomical feature for each slice; extrapolating a 3D volume between adjacent slices based on the respective cross-sectional areas; and computing a total 3D volume of the isolated anatomical feature based on the extrapolated 3D volumes between adjacent slices.

4. The method of claim 1, wherein generating, by the server, physiological information of the isolated 3D surface mesh model associated with the selected pathology comprises: determining a start point and an end point of the isolated anatomical feature and a direction of travel from the start point to the end point; projecting rays at predefined intervals along an axis in at least three directions perpendicular to the direction of travel and determining a distance between an intersection of each projected ray and the 3D surface mesh model; computing a center point at each interval by triangulating the distance between the intersection of each projected ray and the 3D surface mesh model; adjusting the direction of travel at each interval based on direction vectors between adjacent computed center points such that ray projection at the predefined intervals occurs in at least three directions perpendicular to the adjusted direction of travel at each interval; and computing a centerline of the isolated anatomical feature based on the computed center points from the start point to the end point. ​ 5. The method of claim 1, wherein generating, by the server, physiological information associated with the selected pathology for the isolated 3D surface mesh model comprises: computing a centerline of the isolated anatomical feature; determining a start point and an end point of the isolated anatomical feature and a directional vector from the start point to the end point; establishing cut planes at predefined intervals along the centerline based on the directional vector from the start point to the end point, each cut plane being perpendicular to a direction of travel of the centerline at each interval; projecting a ray in the cut plane at each interval to determine a location of an intersection point on the 3D surface mesh model relative to the centerline; and computing a length across the 3D surface mesh model based on the determined location of the intersection point at each interval.

6. The method of claim 1, wherein generating, by the server, physiological information associated with the selected pathology for the isolated 3D surface mesh model comprises: determining a start point and an end point of the isolated anatomical feature; taking slices at predefined intervals along an axis from the start point to the end point; computing a cross-sectional area of each slice bounded by a perimeter of the isolated anatomical feature; and generating a heat map of the isolated anatomical feature based on the cross-sectional area of each slice.

7. The method of claim 1, wherein generating, by the server, physiological information associated with the selected pathology for the isolated 3D surface mesh model comprises: determining a start point and an end point of the isolated anatomical feature; computing a centerline of the isolated anatomical feature; determining directional travel vectors between adjacent points along the centerline; computing a change magnitude of directional travel vectors between adjacent points along the centerline; and generating a heat map of the isolated anatomical feature based on the change magnitude of directional travel vectors between adjacent points along the centerline.

8. The method of claim 1, wherein the generated physiological information associated with the selected pathology for the isolated 3D surface mesh model includes an associated timestamp, the method further comprising: recording, by the server, the generated physiological information and the associated timestamp; and computing, by the server, a change over time between the recorded physiological information based on associated timestamps, which is indicative of a progression of the selected pathology.

9. The method of claim 8, further comprising: computing, by the server, a magnitude of the change over time between the recorded physiological information; and generating, by the server, a heat map of the isolated anatomical feature based on the magnitude of the change over time between the recorded physiological information.

10. The method of claim 1, wherein extracting, by the server, the isolated 3D surface mesh model comprising the patient-specific anatomical feature of the selected pathology from the 3D surface mesh model based on the metadata comprises: isolating an anatomical feature from the one or more classified patient-specific anatomical features based on the selected pathology; analyzing features of the isolated anatomical feature with an anatomical feature database to identify one or more landmarks of the isolated anatomical feature; associating the one or more identified landmarks with the pixels of the medical image; and generating a 3D surface mesh model that defines a surface of the isolated anatomical feature including the identified landmarks.

11. The method of claim 10, further comprising identifying, by the server, a guided trajectory for performing a surgical procedure from a surgical instrument database based on the selected pathology and the one or more identified landmarks; and displaying the guided trajectory to a user.

12. The method of claim 1, further comprising: receiving, by the server, patient demographic data; identifying, by the server, one or more medical devices from a medical device database based on the patient demographic data and the generated physiological information associated with the selected pathology of the isolated 3D surface mesh model; and displaying the identified one or more medical devices to a user.

13. The method of claim 1, further comprising: receiving, by the server, patient demographic data; identifying, by the server, one or more treatment options from a surgical instrument database based on the patient demographic data and the generated physiological information associated with the selected pathology of the isolated 3D surface mesh model; and displaying the identified one or more treatment options to a user.

14. The method of claim 1, wherein extracting, by the server, the isolated 3D surface mesh model of the patient-specific anatomical feature including the selected pathology from the 3D surface mesh model based on the metadata comprises: isolating an anatomical feature from the one or more classified patient-specific anatomical features based on the selected pathology; analyzing features of the isolated anatomical feature with an anatomical feature database to identify one or more landmarks of the isolated anatomical feature; analyzing features of the one or more landmarks with a reference fracture database to detect a fracture of the isolated anatomical feature; and generating a 3D surface mesh model of the isolated anatomical feature including the one or more identified landmarks and the detected fracture.

15. The method of claim 14, further comprising matching the 3D surface mesh model of the isolated anatomical feature to the reference fracture database to classify the detected fracture.

16. The method of claim 1, further comprising: rendering, by the server, the classified one or more patient-specific anatomical features as binary markers; segmenting, by the server, the binary markers into individual anatomical features; and mapping, by the server, the individual anatomical features to raw grayscale values of the medical image and removing background within the medical image, and wherein the generated 3D surface mesh model defines a surface of the individual anatomical features, or includes a volume rendering defined by mapping a particular color or transparency value to the classified one or more patient-specific anatomical features. ​ ​ 17. The method of claim 1, wherein the segmentation algorithm comprises at least one of: a threshold-based, decision tree, chain decision forest, or neural network method.

18. The method of claim 1, wherein the physiological information associated with the selected pathology comprises at least one of: a diameter, a volume, a density, a thickness, a surface area, a Hausdorff unit standard deviation, or a mean value.

19. A system for multi-modal analysis of patient-specific anatomical features from medical images, the system comprising a server and configured to: receive a medical image of a patient and metadata associated with the medical image indicative of a selected pathology; automatically process the medical image using a segmentation algorithm to label pixels of the medical image and generate scores indicative of a likelihood that the pixels are correctly labeled; use an anatomical feature recognition algorithm to probabilistically match associated groups of the labeled pixels to an anatomical knowledge dataset to classify one or more patient-specific anatomical features within the medical image; generate a 3D surface mesh model that bounds surfaces of the one or more classified patient-specific anatomical features; extract an isolated 3D surface mesh model of a patient-specific anatomical feature comprising the selected pathology from the 3D surface mesh model based on the metadata; and generate physiological information of the isolated 3D surface mesh model associated with the selected pathology.

20. The system of claim 19, wherein the isolated 3D surface mesh model comprises a 3D surface mesh model of an anatomical feature isolated from the one or more classified patient-specific anatomical features based on the selected pathology.

21. A non-transitory computer-readable memory medium configured to store instructions thereon, the instructions, when loaded into at least one processor, cause the at least one processor to: receive a medical image of a patient and metadata associated with the medical image indicative of a selected pathology; automatically process the medical image using a segmentation algorithm to label pixels of the medical image and generate scores indicative of a likelihood that the pixels are correctly labeled; use an anatomical feature recognition algorithm to probabilistically match associated groups of the labeled pixels to an anatomical knowledge dataset to classify one or more patient-specific anatomical features within the medical image; generate a 3D surface mesh model that bounds surfaces of the one or more classified patient-specific anatomical features; extract an isolated 3D surface mesh model of a patient-specific anatomical feature comprising the selected pathology from the 3D surface mesh model based on the selected pathology; and generate physiological information of the isolated 3D surface mesh model associated with the selected pathology. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Images