System and method for automated segmentation of patient-specific biostructures for lesion-specific measurements

Abstract

A system and method are provided for multi-scheme analysis of patient-specific anatomical features from medical images. The system may receive a medical image of a patient and metadata associated with the medical image indicative of a selected lesion, and may automatically classify the medical image using a segmentation algorithm. The system may identify one or more patient-specific anatomical features in the medical image using an anatomical feature identification algorithm by searching an anatomical knowledge dataset. A 3D surface mesh model may be generated representing the one or more classified patient-specific anatomical features such that information may be extracted from the 3D surface mesh model based on the selected lesion. Physiological information associated with the selected lesion for the 3D surface mesh model may be generated based on the extracted information.

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims the benefit of priority of GB Patent Application No. 2101908.8, filed on February 11, 2021, the entire contents of which are incorporated herein by reference.

[0002] (Field of use) This disclosure is directed to systems and methods for multi - scheme analysis of patient - specific anatomical features from medical images for lesion - specific measurements for certain use cases in diagnosis, planning, and treatment.

Background Art

[0003] Creating an accurate 3D model of a specific part of a patient's anatomy can help transform surgical techniques by providing insights to clinicians for preoperative planning. Benefits include, for example, better clinical outcomes for the patient, reduced time and cost for surgery, and the patient's ability to better understand the planned surgery.

[0004] However, there is still a need to provide 3D models to provide further insights regarding the patient's anatomy or lesions.

[0005] In light of the aforementioned drawbacks of existing systems and methods, there is a need for improved systems and methods for analyzing a patient's medical images and creating 3D models to assist in diagnosis, planning, and / or treatment.

Summary of the Invention

Means for Solving the Problems

[0006] This disclosure overcomes the shortcomings of previously known systems and methods by providing a system and method for multi-scheme analysis of patient-specific anatomical features from medical images for lesion-specific measurements for specific use cases in diagnosis, planning, and / or treatment.

[0007] The generation of scaled virtual models of a patient's biological structure, such as 3D anatomical models, is an extremely useful tool that can be used to drive personalized, patient-specific decisions in clinical practice, for example, for preoperative planning. This disclosure demonstrates a method for generating patient-specific 3D models of a patient's complete biological structure by, for example, building machine learning models for automatically detecting and segmenting biological structures from medical scans. These models can be trained using curated, semantically labeled data sets. To generate 3D segmentation, a neural network or machine learning algorithm is trained to identify anatomical features within a set of medical images. These images are semantically labeled with the location of the anatomical feature, its component parts, and markers. Thus, the segmentation algorithm can obtain new data sets and their complementary markers, which can then be used to identify new anatomical features or markers.

[0008] The segmentation process is the first step in generating patient-specific insights into anatomical features, which drives decision-making in a clinical setting. The technology, commercially available from Axial Medical Printing Limited (Belfast, United Kingdom), transforms 2D medical scans into scaled 3D models of the patient's biostructure, enabling 3D decision-making and understanding. The output of the segmentation process is a precise set of coordinates representing anatomical features within the scan. This representation of the biostructure allows for a definitive description of the features to be produced, such as standard measurements including size, length, volume, diameter, oblique section, and others. As a result, the shape and location of anatomical features or lesions can be calculated and incorporated into the surgeon's personalized decision-making process. These measurements can be used to drive critical decisions about the patient's condition and any proposed interventions.

[0009] More importantly, the systems described herein can distinguish between normal and pathological states of biological structures and any anatomical features. The training process can be further enhanced with this information, which can then be used to further derive classes of anatomical features. For example, blood can be identified and segmented within a medical scan. By incorporating information about pathological states, blood clots can also be identified and segmented within blood vessels, so that the type and severity of the lesion can be identified. Combined with measured data about biological structures, this information is crucial for decision-making in acute clot-based lesions such as stroke or coronary artery disease.

[0010] Lesion-specific patentable artifacts can be created by combining the automated segmentation algorithms described herein with large, labeled training data sets specific to each lesion, such that the appropriate algorithm and specific data combination create a unique artifact / lesion set. The ability to provide specific groupings of segmentation functionality offers significant benefits to specific clinical problems. Furthermore, the ability to provide automated segmentation also opens up several lesion-specific applications that would benefit from the systems described herein.

[0011] In one aspect, a method is provided for multi-scheme analysis of patient-specific anatomical features from medical images. The method may include: a server receiving medical images of a patient and metadata associated with medical images showing selected lesions; the server automatically processing the medical images using a segmentation algorithm, labeling pixels in the medical images, and generating a score indicating the likelihood that the pixels are correctly labeled; the server probabilistically matching associated groups of labeled pixels to a set of anatomical knowledge data classifying one or more patient-specific anatomical features in the medical images using an anatomical feature recognition algorithm; the server generating a 3D surface mesh model defining the surface of one or more classified patient-specific anatomical features; the server extracting information from the 3D surface mesh model based on selected lesions; and the server generating physiological information associated with the selected lesions for the 3D surface mesh model based on the extracted information. For example, the information extracted from the 3D surface mesh model may include a 3D surface mesh model of anatomical features separated from one or more classified patient-specific anatomical features based on selected lesions.

[0012] The server may generate physiological information associated with a selected lesion for a 3D surface mesh model, which may include determining the start and end points of the isolated anatomical feature, obtaining slices at predetermined intervals along the axis from start to end, calculating the cross-sectional area of ​​each slice defined by the periphery of the isolated anatomical feature, extrapolating the 3D volume between adjacent slices based on each cross-sectional area, and calculating the overall 3D volume of the isolated anatomical feature based on the extrapolated 3D volume between adjacent slices.

[0013] Generating physiological information associated with a selected lesion for a 3D surface mesh model by the server may include determining the start and end points of the isolated anatomical feature and the direction of progression from the start to the end point; projecting rays along the axis in at least three directions perpendicular to the direction of progression at predetermined intervals and determining the distance between each projected ray and the intersection point with the 3D surface mesh model; calculating the center point at each interval by triangulation of the distance between each projected ray and the intersection point with the 3D surface mesh model; adjusting the direction of progression at each predetermined interval based on the direction vector between adjacent calculated center points so that ray projections at each interval occur in at least three directions perpendicular to the adjusted direction of progression at each interval; and calculating the centerline of the isolated anatomical feature based on the calculated center point from the start to the end point.

[0014] Generating physiological information associated with a selected lesion for a 3D surface mesh model by a server may include: calculating the centerline of the isolated anatomical feature; determining the start and end points of the isolated anatomical feature and the direction vector from the start to the end point; establishing cross-sections along the centerline at predetermined intervals based on the direction vector from the start to the end point, with each cross-section perpendicular to the direction of progression of the centerline at each interval; projecting rays onto the cross-section at each interval to determine the position of the intersection point on the 3D surface mesh model from the centerline; and calculating the length traversed across the 3D surface mesh model based on the determined position of the intersection point at each interval.

[0015] The server may generate physiological information associated with selected lesions for a 3D surface mesh model, which may include determining the start and end points of isolated anatomical features, obtaining slices at predetermined intervals along the axis from start to end, calculating the cross-sectional area of ​​each slice defined by the periphery of the isolated anatomical features, and generating a heatmap of the isolated anatomical features based on the cross-sectional area of ​​each slice.

[0016] The server may generate physiological information associated with selected lesions for a 3D surface mesh model, which may include determining the start and end points of isolated anatomical features, calculating the midline of the isolated anatomical features, determining the directional progression vectors between adjacent points along the midline, calculating the magnitude of the change in the directional progression vectors between adjacent points along the midline, and generating a heatmap of the isolated anatomical features based on the magnitude of the change in the directional progression vectors between adjacent points along the midline.

[0017] In some embodiments, the generated physiological information associated with a selected lesion for a 3D surface mesh model may include an associated timestamp, thereby the method further includes recording the generated physiological information and the associated timestamp by a server, and calculating the change over time between the recorded physiological information indicating the progression of the selected lesion, based on the associated timestamp by the server. Thus, the method may further include calculating the magnitude of the change over time between the recorded physiological information by a server, and generating a heatmap of isolated anatomical features, based on the magnitude of the change over time between the recorded physiological information by the server.

[0018] Extracting information from a 3D surface mesh model based on a selected lesion by the server may include: separating anatomical features from one or more classified patient-specific anatomical features based on the selected lesion; analyzing the features of the separated anatomical features using an anatomical feature database to identify one or more markers of the separated anatomical features; associating one or more identified markers with pixels in a medical image; and generating a 3D surface mesh model that defines the surface of the separated anatomical features having the identified markers. Furthermore, the method may further include the server identifying guide trajectories for performing surgical procedures from a surgical implementation database based on the selected lesion and one or more identified markers; and displaying the guide trajectories to the user.

[0019] In addition, the method may further include, by a server, receiving patient demographic data; by the server, identifying one or more medical devices from a medical device database based on the patient demographic data and generated physiological information associated with selected lesions for a 3D surface mesh model; and displaying the identified one or more medical devices to the user. Furthermore, the method may further include, by a server, receiving patient demographic data; by the server, identifying one or more treatment options from a surgical implementation database based on the patient demographic data and generated physiological information associated with selected lesions for a 3D surface mesh model; and displaying the identified one or more treatment options to the user.

[0020] Extracting information from a 3D surface mesh model based on a selected lesion by the server may include: separating an anatomical feature from one or more classified patient-specific anatomical features based on the selected lesion; analyzing the features of the separated anatomical features using an anatomical feature database to identify one or more markers of the separated anatomical features; analyzing the features of one or more markers using a reference fracture database to detect fractures of the separated anatomical features; and generating a 3D surface mesh model of the separated anatomical features comprising one or more identified markers and the detected fractures. Therefore, the method may further include matching the 3D surface mesh model of the separated anatomical features against the reference fracture database and classifying the detected fractures.

[0021] The method may further include the server depicting one or more classified patient-specific anatomical features within binary labels, the server separating the binary labels into distinct anatomical features, and the server mapping the distinct anatomical features to the original grayscale values ​​of the medical image and removing the background within the medical image, wherein the generated 3D surface mesh model features volume rendering defined by defining the surface of the distinct anatomical features or by mapping specific color or transparency values ​​to one or more classified patient-specific anatomical features. In some embodiments, the segmentation algorithm may include at least one of threshold-based, decision tree, chained decision forest, or neural network methods. Physiological information associated with selected lesions may include at least one of diameter, volume, density, thickness, surface area, Hounsfield unit standard deviation, or mean.

[0022] According to another aspect of this disclosure, a system for multi-scheme analysis of patient-specific anatomical features from medical images is provided. The system may include a server that receives medical images of a patient and metadata associated with the medical images showing selected lesions, automatically processes the medical images using a segmentation algorithm, labels pixels in the medical images, generates a score indicating the likelihood that the pixels are correctly labeled, probabilistically matches associated groups of labeled pixels against a set of anatomical knowledge data using an anatomical feature recognition algorithm, classifies one or more patient-specific anatomical features in the medical images, generates a 3D surface mesh model that defines the surface of one or more classified patient-specific anatomical features, extracts information from the 3D surface mesh model based on the selected lesions, and generates physiological information associated with the selected lesions for the 3D surface mesh model based on the extracted information. For example, the information extracted from the 3D surface mesh model may include a 3D surface mesh model of anatomical features separated from one or more classified patient-specific anatomical features based on the selected lesions.

[0023] According to yet another aspect of the present disclosure, a non-transitory computer-readable memory medium having instructions stored thereon is provided, and the instructions, when loaded by at least one processor, cause the at least one processor to receive a medical image of a patient and metadata associated with the medical image indicating a selected lesion, automatically process the medical image using a segmentation algorithm, label the pixels of the medical image, generate a score indicating the likelihood that the pixels are correctly labeled, probabilistically match the associated groups of labeled pixels to an anatomical knowledge data set using an anatomical feature identification algorithm, classify one or more patient-specific anatomical features within the medical image, generate a 3D surface mesh model defining the surface of the one or more classified patient-specific anatomical features, extract information from the 3D surface mesh model based on the selected lesion, and generate physiological information associated with the selected lesion for the 3D surface mesh model based on the extracted information. The present invention provides, for example, the following: (Item 1) A method for multi-scheme analysis of patient-specific anatomical features from medical images, wherein the method is: The server receives the patient's medical image and metadata associated with the medical image that shows the selected lesion. The server automatically processes the medical image using a segmentation algorithm, labels the pixels of the medical image, and generates a score indicating the likelihood that the pixels are correctly labeled. The server uses an anatomical feature recognition algorithm to probabilistically match the associated groups of the labeled pixels against a set of anatomical knowledge data, thereby classifying one or more patient-specific anatomical features in the medical image. The server generates a 3D surface mesh model that defines the surface of one or more classified patient-specific anatomical features, The server extracts information from the 3D surface mesh model based on the selected lesion, [[ID=​​​​​​​​​​​​​​​​​​​​​​​​ The server generates physiological information associated with the selected lesion for the 3D surface mesh model, Determining the starting and ending points of the separated anatomical features, and the direction of progression from the starting point to the ending point, Projecting rays in at least three directions perpendicular to the direction of travel at predetermined intervals along the axis, and determining the distance between each projected ray and the intersection point of the 3D surface mesh model, The center point is calculated by triangulating the distance between each projected ray and the intersection point of the 3D surface mesh model at each interval, At each interval, the direction of propagation is adjusted based on the direction vector between adjacent calculated center points, thereby causing ray projection at the predetermined interval to occur in at least three directions perpendicular to the adjusted direction of propagation at each interval. Based on the calculated center point from the starting point to the ending point, the centerline of the separated anatomical features is calculated. The method described in item 2, including the method described in item 2. (Item 5) The server generates physiological information associated with the selected lesion for the 3D surface mesh model, Calculating the midline of the separated anatomical features, Determining the start and end points of the separated anatomical features, and the direction vector from the start point to the end point, Based on the direction vector from the starting point to the ending point, a cross-section is established at predetermined intervals along the center line, wherein each cross-section is perpendicular to the direction of travel of the center line at each interval. Projecting light rays onto the cross-section at each interval and determining the position of the intersection point on the 3D surface mesh model from the center line, Based on the determined intersection points at each of the aforementioned intervals, the length across the 3D surface mesh model is calculated. The method described in item 2, including the method described in item 2. (Item 6) The server generates physiological information associated with the selected lesion for the 3D surface mesh model, Determining the start and end points of the separated anatomical features, Obtaining slices at predetermined intervals along the axis from the starting point to the ending point, Calculate the cross-sectional area of ​​each slice defined by the periphery of the separated anatomical features, Based on the cross-sectional area of ​​each slice, a heatmap of the separated anatomical features is generated. The method described in item 2, including the method described in item 2. (Item 7) The server generates physiological information associated with the selected lesion for the 3D surface mesh model, Determining the start and end points of the separated anatomical features, Calculating the midline of the separated anatomical features, Determining the directional vector between adjacent points along the aforementioned center line, The magnitude of the change in the directional vector between adjacent points along the aforementioned center line is calculated, Based on the magnitude of the change in the directional propagation vector between adjacent points along the center line, a heat map of the separated anatomical features is generated. The method described in item 2, including the method described in item 2. (Item 8) The generated physiological information associated with the selected lesion for the 3D surface mesh model includes an associated timestamp, and the method The server records the generated physiological information and the associated timestamp. The server calculates, based on associated timestamps, the temporal changes between the recorded physiological information indicating the progression of the selected lesion. The method described in item 2, further including the method described in item 2. (Item 9) The server calculates the magnitude of the time-dependent changes between the recorded physiological information, The server generates a heatmap of the separated anatomical features based on the magnitude of the temporal changes between the recorded physiological information. The method described in item 8, further including the method described in item 8. (Item 10) The server extracts information from the 3D surface mesh model based on the selected lesion. Based on the selected lesions, the anatomical features are separated from the one or more classified patient-specific anatomical features. Using an anatomical feature database, the characteristics of the separated anatomical features are analyzed, and one or more markers of the separated anatomical features are identified. Associating one or more identified markers with the pixels of the medical image, To generate a 3D surface mesh model that defines the surface of the separated anatomical features having the identified markers, The method described in item 1, including the method described in item 1. (Item 11) The server identifies a guide trajectory for performing a surgical procedure from a surgical implementation database based on the selected lesion and the one or more identified landmarks. Displaying the aforementioned guidance path to the user The method described in item 10, further including the method described in item 10. (Item 12) The aforementioned server receives patient background data, The server identifies one or more medical devices from a medical device database based on the patient background data and the generated physiological information associated with the selected lesion for the 3D surface mesh model. Displaying one or more identified medical devices to the user The method described in item 1, further including the method described in item 1. (Item 13) The aforementioned server receives patient background data, The server identifies one or more treatment options from a surgical implementation database based on the patient background data and the generated physiological information associated with the selected lesion for the 3D surface mesh model. Displaying one or more of the identified treatment options to the user The method described in item 1, further including the method described in item 1. (Item 14) The server extracts information from the 3D surface mesh model based on the selected lesion. Based on the selected lesions, the anatomical features are separated from the one or more classified patient-specific anatomical features. Using an anatomical feature database, the characteristics of the separated anatomical features are analyzed, and one or more markers of the separated anatomical features are identified. Using a reference fracture database, the characteristics of one or more of the aforementioned landmarks are analyzed to detect fractures of the separated anatomical features, To generate a 3D surface mesh model of the isolated anatomical features comprising the one or more identified landmarks and the detected fractures. The method described in item 1, including the method described in item 1. (Item 15) The method of item 14, further comprising matching the 3D surface mesh models of the separated anatomical features against the reference fracture database and classifying the detected fractures. (Item 16) The server then depicts one or more of the classified patient-specific anatomical features in binary labels, The server separates the binary marker into separate anatomical features, The server maps the separate anatomical features to the original grayscale values ​​of the medical image and removes the background within the medical image. It further includes, The method according to item 1, wherein the generated 3D surface mesh model comprises volume rendering defined by defining the surface of the distinct anatomical features or by mapping specific color or transparency values ​​to one or more classified patient-specific anatomical features. (Item 17) The method according to item 1, wherein the segmentation algorithm comprises at least one of threshold-based, decision tree, chained decision forest, or neural network methods. (Item 18) The method according to item 1, wherein the physiological information associated with the selected lesion comprises at least one of the following: diameter, volume, density, thickness, surface area, Hounsfield unit standard deviation, or mean. (Item 19) A system for multi-scheme analysis of patient-specific anatomical features from medical images, wherein the system comprises a server, and the server is Receiving a patient's medical image and metadata associated with the medical image showing a selected lesion, Using a segmentation algorithm, the medical image is automatically processed, the pixels of the medical image are labeled, and a score is generated indicating the likelihood that the pixels are correctly labeled. Using an anatomical feature recognition algorithm, the associated groups of labeled pixels are probabilistically matched against a set of anatomical knowledge data to classify one or more patient-specific anatomical features in the medical image. To generate a 3D surface mesh model that defines the surface of one or more classified patient-specific anatomical features, Based on the selected lesion, information is extracted from the 3D surface mesh model. Based on the extracted information, physiological information associated with the selected lesion for the 3D surface mesh model is generated. A system configured to perform the following actions. (Item 20) The system according to item 19, wherein the information extracted from the 3D surface mesh model comprises a 3D surface mesh model of anatomical features separated from the one or more classified patient-specific anatomical features based on the selected lesion. (Item 21) A non-transient computer-readable memory medium, wherein the memory medium is configured to store instructions thereon, and the instructions, once loaded by at least one processor, Receiving a patient's medical image and metadata associated with the medical image showing a selected lesion, Using a segmentation algorithm, the medical image is automatically processed, the pixels of the medical image are labeled, and a score is generated indicating the likelihood that the pixels are correctly labeled. Using an anatomical feature recognition algorithm, the associated groups of labeled pixels are probabilistically matched against a set of anatomical knowledge data to classify one or more patient-specific anatomical features in the medical image. To generate a 3D surface mesh model that defines the surface of one or more classified patient-specific anatomical features, Based on the selected lesion, information is extracted from the 3D surface mesh model. Based on the extracted information, physiological information associated with the selected lesion for the 3D surface mesh model is generated. A non-transient, computer-readable memory medium that causes at least one processor to perform the above. [Brief explanation of the drawing]

[0024] [Figure 1] Figure 1 shows some exemplary components that may be included within a multi-scheme analysis platform based on the principles of this disclosure.

[0025] [Figure 2] Figure 2 is a flowchart illustrating exemplary method steps for multi-scheme analysis of patient-specific anatomical features from medical images according to the principles of this disclosure.

[0026] [Figure 3] Figure 3 is a flowchart illustrating exemplary method steps for generating volumetric measurements of patient-specific anatomical features according to the principles of this disclosure.

[0027] [Figure 4] Figure 4A illustrates cross-sectional area measurements at various points along a blood vessel according to the principle of this disclosure, and Figure 4B illustrates the determination of volume based on the cross-sectional area measurements.

[0028] [Figure 5] Figure 5 is a flowchart illustrating exemplary method steps for generating patient-specific anatomical feature centerline measurements according to the principles of this disclosure.

[0029] [Figure 6] Figure 6 illustrates the determination of the centerline according to the principles of this disclosure.

[0030] [Figure 7] Figure 7A illustrates the center point of the blood vessel, Figure 7B illustrates the centerline of the blood vessel, Figure 7C illustrates the measured length of the centerline of the blood vessel, and Figure 7D illustrates the blood vessel as depicted across the medical image.

[0031] [Figure 8] Figure 8A illustrates the start and end points of patient-specific anatomical features, Figure 8B illustrates the midline of patient-specific anatomical features, Figure 8C illustrates the midline of various patient-specific anatomical features, and Figure 8D illustrates the midline of a network of patient-specific anatomical features.

[0032] [Figure 9] Figure 9 is a flowchart illustrating exemplary method steps for generating patient-specific anatomical feature surface length measurements according to the principles of this disclosure.

[0033] [Figure 10] Figure 10 illustrates the determination of the surface length according to the principle of this disclosure.

[0034] [Figure 11] Figure 11 illustrates the surface length of patient-specific anatomical features.

[0035] [Figure 12] Figure 12 is a flowchart illustrating exemplary method steps for generating a heatmap of patient-specific anatomical features based on volume according to the principles of this disclosure.

[0036] [Figure 13] Figures 13A and 13B illustrate volume-based heatmaps of patient-specific anatomical features.

[0037] [Figure 14] Figure 14 is a flowchart illustrating exemplary method steps for generating a heatmap of patient-specific anatomical features based on the degree of meandering according to the principles of this disclosure.

[0038] [Figure 15] Figure 15 illustrates a heatmap based on tortuosity of patient-specific anatomical features.

[0039] [Figure 16] Figure 16 is a flowchart illustrating exemplary method steps for generating a 3D surface mesh model of patient-specific anatomical features using identified markers according to the principles of this disclosure.

[0040] [Figure 17A] Figure 17A illustrates exemplary method steps for mapping identified anatomical features specific to a patient onto a 3D surface mesh model according to the principles of this disclosure.

[0041] [Figure 17B] Figure 17B illustrates identified landmarks of patient-specific anatomical features mapped onto a 3D surface mesh model.

[0042] [Figure 18] Figure 18 is a flowchart illustrating exemplary method steps for identifying medical devices and treatment options for lesions according to the principles of this disclosure.

[0043] [Figure 19] Figure 19A illustrates a bone lesion, and Figures 19B and 19C illustrate various medical devices that may be used to treat the lesion.

[0044] [Figure 20] Figure 20 is a flowchart illustrating exemplary method steps for detecting and classifying patient-specific anatomical features according to the principles of this disclosure.

[0045] [Figure 21] Figures 21A-21D illustrate the steps of mapping detected patient-specific anatomical features onto a 3D surface mesh model using the principles of this disclosure.

[0046] [Figure 22] Figure 22 is a flowchart illustrating exemplary method steps for tracking the temporal progression of a lesion according to the principles of this disclosure.

[0047] [Figure 23] Figures 23A-23F illustrate the various stages of disease progression over time.

[0048] [Figure 24] Figures 24A-24F illustrate heatmaps of various stages of lesion progression over time.

[0049] [Figure 25] Figure 25 is a flowchart illustrating exemplary method steps for analyzing the physiological parameters of distinct anatomical features according to the principles of this disclosure.

[0050] [Figure 26] Figure 26 illustrates the generation of 3D volume rendering of separate anatomical features according to the principles of this disclosure.

[0051] [Figure 27] Figure 27A illustrates the original medical image of a specific anatomical feature of a patient, Figure 27B illustrates the distinct anatomical feature overlaid on the original medical image, Figure 27C illustrates the distinct anatomical feature with the background removed, and Figure 27D illustrates a 3D volume rendering of the distinct anatomical feature.

[0052] [Figure 28A] Figures 28A and 28B illustrate exemplary method steps for measuring occlusion of patient-specific anatomical features according to the principles of this disclosure. [Figure 28B] Figures 28A and 28B illustrate exemplary method steps for measuring occlusion of patient-specific anatomical features according to the principles of this disclosure.

[0053] [Figure 29] Figure 29 is a flowchart illustrating exemplary method steps for analyzing the physiological parameters of distinct anatomical features according to the principles of this disclosure.

[0054] [Figure 30] Figures 30A–30E illustrate the steps for generating patient-specific anatomical feature measurements according to the principles of this disclosure.

[0055] [Figure 31] Figure 31 illustrates the effect on the weighted mask and loss function generated using the Euclidean distance-weighted approach based on the principles of this disclosure.

[0056] [Figure 32] Figure 32 illustrates various segments of bone in medical images for training purposes based on the principles of this disclosure.

[0057] [Figure 33] Figure 33 illustrates various segments of myocardium in medical images of ground truth data for training purposes according to the principles of this disclosure. [Modes for carrying out the invention]

[0058] Referring to Figure 1, the components that may be included within the multi-scheme analysis platform 100 are described. The platform 100 may include one or more processors 102, a communication network 104, a power source 106, a user interface 108, and / or memory 110. One or more electrical components and / or circuits may perform some or all of the roles of the various components described herein. Although described separately, it should be understood that electrical components do not have to be separate structural elements. For example, the platform 100 and the communication network 104 may be embodied within a single chip. In addition, although the platform 100 is described as having memory 110, the memory chip may be provided separately.

[0059] Platform 100 may include memory and / or be coupled to memory via one or more buses, from which information can be read or written. Memory 110 may include a processor cache including a multi-level hierarchical cache with different levels having different capacities and access speeds. Memory may also include random access memory (RAM), other volatile storage devices, or non-volatile storage devices. Memory 110 may be RAM, ROM, flash, other volatile or non-volatile storage devices, or other known memories, or any combination thereof, and preferably includes storage from which data can be selectively stored. For example, storage devices may include, for example, hard drives, optical disks, flash memory, and Zip drives. Programmable instructions may be stored on memory 110 to execute algorithms for automatically segmenting and identifying patient-specific anatomical features in medical images, including corresponding anatomical landmarks; generating a 3D surface mesh model of the patient-specific anatomical features; and extracting information from the 3D surface mesh model to generate physiological information of the patient-specific anatomical features based on selected lesions.

[0060] Platform 100 may incorporate a processor 102, which may consist of one or more processors, which may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, separate gate or transistor logic, separate hardware components, or any preferred combination thereof, designed to perform the functions described herein. Platform 100 may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0061] Platform 100, in conjunction with firmware / software stored in memory, can run an operating system such as Windows®, Mac OS, Unix®, or Solaris® 5.10 (e.g., operating system 124). Platform 100 also runs software applications stored in memory. For example, the software may be a program in any suitable programming language known to those skilled in the art, including, for example, C++, PHP, or Java®.

[0062] The communication network 104 may include a network that enables platform 100 to communicate with an image acquisition device and / or with other computing devices to receive image files (e.g., 2D medical images) and metadata associated with them that indicate patient-specific lesions. In addition, or alternatively, image files may be uploaded directly to platform 100. The communication network 104 may be configured for wired and / or wireless communication via the Internet, a telephone network, a Bluetooth® network, and / or a Wi-Fi network, etc., using techniques known in the art. The communication network 104 may be a communication chip known in the art, such as a Bluetooth® chip and / or a Wi-Fi chip. The communication network 104 enables platform 100 to transfer information such as a 3D surface mesh model, physiological measurements, and information such as treatment options, local and / or remote locations such as servers.

[0063] The power source 106 may supply alternating current or direct current. In the direct current embodiment, the power source may include a suitable battery, such as a replaceable battery or a rechargeable battery, and the device may include a circuit for charging the rechargeable battery and a detachable power cord. The power source 106 may be charged by a charger, via an inductive coil in the charger and an inductive coil. Alternatively, the power source 106 may be a port that allows the platform 100 to be plugged into a conventional wall socket, for example, via a cord with an AC / DC power converter and / or a USB port, to power components within the platform 100.

[0064] The user interface 108 may be used to receive input from the user and / or provide output to the user. For example, the user interface 108 may include a touchscreen, display, switch, dial, light, etc. Thus, the user interface 108 may display information such as a 3D surface mesh model, physiological measurements, heatmaps, a list of available medical devices for patient-specific lesions, and treatment options, and may facilitate diagnosis, preoperative planning, and treatment for specific use cases, as will be described in more detail below. Furthermore, the user interface 108 may receive user input, including patient background data such as patient size, age, weight, medical history, and patient-specific lesions, and user feedback based on the information displayed to the user, such as corrected anatomical feature identification, physiological measurements, and specific anatomical feature selection, so that the platform 100 can adjust the information as appropriate. In some embodiments, the user interface 108 is not located on the platform 100 but is instead provided on a remote external computing device that is communicably connected to the platform 100 via a communication network 104.

[0065] Memory 110, an example of a non-transient computer-readable medium, may be used to store an operating system (OS) 124, an image receiver module 112, a segmentation module 114, an anatomical feature identification module 116, a 3D surface mesh model generation module 118, an anatomical feature information extraction module 120, and a physiological information generation module 122. The modules are provided in the form of computer-executable instructions that can be executed by a processor 102 to perform the various operations according to this disclosure.

[0066] The image receiver module 112 may be operated by the processor 102 to receive standard medical images obtained from one or a combination of CT, MRI, PET, and / or SPCET scanners, e.g., 2D and / or 3D medical images of one or more patient-specific anatomical features. The medical images may be formatted in a standard-compliant format using DICOM, etc. The medical images may include metadata embedded therein indicating patient-specific lesions associated with patient-specific anatomical features within the medical images. The image receiver module 112 may preprocess the medical images for further processing and analysis, as will be described in more detail below. For example, the medical images may be preprocessed to generate a new set of medical images, which are uniformly distributed according to a predetermined orientation based on patient-specific anatomical features, specific lesions of the patient, or any downstream use such as preoperative training and / or machine learning / neural network training purposes. Furthermore, the image receiver module 112 may receive medical images obtained simultaneously from multiple viewpoints of patient-specific anatomical features to improve the segmentation of patient-specific anatomical features.

[0067] The segmentation module 114 is operated by the processor 102 for automated segmentation of medical images received by the image receiver module 112, and may, for example, assign labels to each pixel of the medical image. The assigned labels may represent specific tissue types, such as bone, soft tissue, blood vessels, organs, etc. Specifically, the segmentation module 114 may use machine learning-based image segmentation techniques, including one or a combination thereof, of the following techniques: threshold-based, decision tree, chained decision forest, or neural network methods, so that the results of each technique can be combined to produce a final segmented result, as described in Haslam's U.S. Patent No. 11,138,790 and U.S. Patent Application Publication No. 2021 / 0335041 (both assigned to the assignees of this disclosure, and both are incorporated herein by reference). The machine learning-based image segmentation techniques may be trained using a knowledge database containing pre-labeled medical images (i.e., ground truth data).

[0068] For example, the segmentation module 114 may apply a first segmentation technique, such as threshold-based segmentation, to assign a label to each pixel of a medical image based on a characteristic, such as whether the pixel's Hounsfield value meets or exceeds a predetermined threshold. The predetermined threshold may be determined, for example, through histogram analysis, as described in U.S. Patent No. 11,138,790. The segmentation module 114 may further extend the threshold-based segmentation technique by using a logistic or stochastic function to calculate a score regarding the likelihood that a pixel is of a tissue type that would be labeled by threshold-based segmentation.

[0069] The segmentation module 114 then applies a decision tree to each labeled pixel of the medical image, thereby classifying / labeling each pixel at least partially based on (but not solely on) a score. As described in U.S. Patent No. 11,138,790, the decision tree may be applied to a portion of the labeled pixels by subsampling the medical image, thereby allowing the segmentation module 114 to reconstruct the full segmentation of the medical image by using standard interpolation methods to upscale the labeled pixels of a portion of the pixels in the medical image. For each pixel, the decision tree may consider not only a score but also, for example, the following properties: the number of pixels in its vicinity that look approximately like bone; the number of pixels in its vicinity that look exactly like bone; or the intensity of the overall gradient of the image at a given pixel. For example, if a pixel is labeled as bone with a score of 60 / 100, the first decision node of the decision tree may ask for the number of pixels in its vicinity that look approximately like bone. If the answer is close to zero, meaning there are very few pixels in the vicinity of the pixel that look almost like bone, the segmentation module 114 may determine that the pixel is not bone, even if the previous bone label had a score of 60 / 100. A new score may then be generated regarding the likelihood that the pixel was correctly labeled by the decision tree algorithm. Thus, applying a decision tree to pixels in a medical image can produce a more accurate final segmentation result with less noise. As will be understood by those skilled in the art, the decision tree may also take into account other properties that may be useful in determining the label for a pixel.

[0070] In addition, or alternatively, the segmentation module 114 may apply a chained decision forest, in which the results of the initial / previous decision tree and the results of another segmentation technique (e.g., a neural network) for the same pixel can be fed into a new decision tree along with the scores associated with the results. For example, the new decision tree may ask one or more questions, as described above, to determine whether each of the previous segmentation techniques correctly labeled the pixel. Thus, if the initial / previous decision tree labeled the pixel as bone, but the neural network labeled it as not bone, the new decision tree may determine that the pixel is bone based on its responses to one or more questions asked by the chained decision forest, thereby discarding the label assigned by the neural network for that pixel. Furthermore, each forest node may be treated as a simple classifier that generates a score regarding the likelihood that the pixel was correctly labeled by each subsequent new decision tree. Therefore, applying a chained decision forest to pixels in medical images may produce more accurate final segmentation results.

[0071] The anatomical feature recognition module 116 may be executed by the processor 102 to identify one or more patient-specific anatomical features in a medical image by probabilistically matching pixels labeled by the segmentation module 114 against a set of anatomical knowledge data in a knowledge database. Specifically, as described in U.S. Patent No. 11,138,790, the anatomical feature recognition module 116 may first group pixels labeled by the segmentation module 114 by establishing links between different labeled / classified pixels, for example, based on similarity between the labeled pixels. For example, all pixels labeled as "bone" may be grouped / linked together in a first group, all pixels labeled as "organ" may be grouped / linked together in a second group, and all pixels labeled as "blood vessel" may be grouped / linked together in a third group.

[0072] The anatomical feature recognition module 116 can then use an anatomical feature recognition algorithm to search an anatomical knowledge data set and identify patient-specific anatomical features in a medical image by establishing links between grouped labeled pixels using existing knowledge within the anatomical knowledge data set. For example, existing knowledge may include known information about various anatomical features such as tissue types (e.g., bone, blood vessels, or organs) represented as nodes in a graph database of the anatomical knowledge data set, and pre-labeled ground truth data that can be used to train various segmentation algorithms.

[0073] A medical ontology of existing knowledge of anatomical features in a graph database may be represented as a series of nodes, which are grouped together through at least one of the following: function, proximity, anatomical grouping, or frequency of expression within the same medical image scan. For example, nodes representing organs may be grouped together as the heart because they are within a predetermined proximity to each other, and all of them are grouped together as the aorta, and they are neighboring nodes representing blood vessels because they have a high frequency of expression within the same medical image scan. Thus, an anatomical feature recognition algorithm may identify patient-specific anatomical features in a medical image by, for example, searching the graph database to determine the group of nodes most similar to a grouped labeled pixel based on established links between the grouped labeled pixel and the group of nodes. The anatomical feature recognition module 116 may further generate a score representing the likelihood that the patient-specific anatomical feature was correctly identified by the anatomical feature recognition algorithm.

[0074] The 3D surface mesh model generation module 118 may be executed by processor 102 to generate a 3D surface mesh model of patient-specific anatomical features in a medical image, based on the results of the segmentation algorithm and anatomical feature recognition algorithm described above, and to extract the 3D surface mesh model from a scalar volume to generate a 3D printable model. For example, as described in U.S. Patent No. 11,138,790, the 3D surface mesh model may have the following properties: all individual surfaces are closed manifolds; appropriate support is used to keep the individual surfaces / volumes in place; appropriate support is used to facilitate 3D printing; and / or the surface volume is not hollow so that the 3D surface mesh model is 3D printable. Furthermore, the 3D surface mesh model generation module 118 may generate a 3D surface mesh model of patient-specific anatomical features, including any corresponding markers of anatomical features, as described in more detail below.

[0075] The anatomical feature information extraction module 120 may be executed by the processor 102 to extract information from the 3D surface mesh model generated by the 3D surface mesh model generation module 118. For example, the anatomical feature information extraction module 120 may extract one or more specific anatomical features from the 3D surface mesh model representing patient-specific anatomical features in the medical image, based on selected lesions indicated in metadata received by the image receiver module 112. Alternatively, the platform 100 may receive information directly from the user, via the user interface 108, indicating selected lesions associated with the medical image, for example, along with patient background data and medical history. Therefore, if a particular lesion is known in relation to a given patient, the anatomical feature information extraction module 120 may automatically extract the 3D surface mesh model of the specific anatomical feature containing the lesion from the 3D surface mesh model generated by the 3D surface mesh model generation module 118.

[0076] The physiological information generation module 122 may be executed by the processor 102 to generate physiological information associated with a selected lesion for a 3D surface mesh model, based on the information extracted by the anatomical feature information extraction module 120. For example, based on the selected lesion, the physiological information generation module 122 may perform calculations to determine physiological measurements relevant to the diagnosis and / or treatment of the lesion by providing a list of appropriate medical devices and / or treatment options for treating the lesion, for example, based on anatomical feature measurements and patient background data. The list of medical devices and / or treatment options may be extracted by the physiological information generation module 122 from a medical device database or a surgical procedure implementation database. As will be further described below with reference to Figure 3A-24F, the physiological measurements associated with the selected lesion determined by the physiological information generation module 122 may include, but are not limited to, volume, cross-sectional area, diameter, centerline, surface, density, thickness, tortuosity, feature size and location, blood clot, occlusion, and the rate of growth over time of anatomical features and / or corresponding landmarks. Furthermore, the physiological information generated by the physiological information generation module 122 can be used to generate a heat map to facilitate the visual observation of physiological measurements of patient-specific anatomical features.

[0077] Referring here to Figure 2, an exemplary method 200 for multi-scheme analysis of patient-specific anatomical features from medical images is provided using platform 100. In step 202, medical images and metadata associated with medical images showing selected lesions may be received by image receiver module 112. As described above, information showing selected lesions may be received directly via user input, along with patient background data. In step 204, segmentation module 114 may automatically process the medical images using a segmentation algorithm, label the pixels of the medical images, and generate a score indicating the likelihood that the pixels were correctly labeled. For example, the segmentation algorithm may use one or a combination of various machine learning-based image segmentation techniques trained on a set of knowledge data of pre-labeled medical images to label the pixels of the medical images.

[0078] In step 206, the anatomical feature recognition module 116 may group the pixels labeled in step 204 together based on similarity and use an anatomical feature recognition algorithm to probabilistically match the associated groups of labeled pixels against a set of anatomical knowledge data to classify one or more patient-specific anatomical features in the medical image. In step 208, the 3D surface mesh model generation module 118 may generate a 3D surface mesh model that defines the surface of one or more classified patient-specific anatomical features in the medical image. In step 210, the anatomical feature information extraction module 120 may extract information from the 3D surface mesh model based on the selected lesion, and the physiological information generation module 122 may generate physiological information associated with the selected lesion for the 3D surface mesh model based on the extracted information. The generated physiological information is described in more detail below with reference to Figure 3A-24F.

[0079] Referring here to Figure 3, an exemplary method 300 for generating volumetric measurements of patient-specific anatomical features is provided. As described above with respect to step 210 of method 200 for multi-scheme analysis of patient-specific anatomical features from medical images in Figure 2, physiological information of patient-specific anatomical features associated with selected lesions, e.g., volumetric measurements, can be generated from the generated 3D surface mesh model. For example, in step 302, specific anatomical features may be separated from patient-specific anatomical features for further analysis, e.g., based on selected lesions in the medical images, such as indicated by metadata associated with the medical images, thereby the 3D surface mesh model of the separated anatomical features can be extracted and recorded from the 3D surface mesh model of patient-specific anatomical features. Thus, only anatomical features with lesions can be further analyzed to generate physiological information associated with the lesions.

[0080] In step 304, the start and end points of the isolated anatomical feature are determined, for example, at opposite ends of the isolated anatomical feature. For example, the start and end points may be determined via a machine learning algorithm that searches an anatomical knowledge data set and derives the start and end points of the isolated anatomical feature. In step 306, a predetermined step size may be determined so that slices can be obtained at regular intervals defined by the predetermined step size along the axis of the isolated anatomical feature. For example, the axis may be the centerline of the isolated anatomical feature, determined based on a direction vector extending from the start point to the end point, as will be described in more detail below. Thus, slices of the isolated anatomical feature can be obtained at each interval perpendicular to the direction of progression along the centerline, starting from the start point toward the end point.

[0081] In step 308, the cross-sectional area in each slice of the isolated anatomical feature can be calculated using a standard calculation function, such that it is defined by the periphery of the isolated anatomical feature, as shown in Figure 4A. For example, the cross-sectional area of ​​an automatically segmented marker for a specific part of a biostructure, such as the mitral valve or aortic valve biostructure, can be calculated based on the derivative of the two largest cross-sections of the biostructure, for example, using A × B × π. In the case of an aneurysm, this data can be used to automatically provide the surgeon with the aneurysm height / aneurysm neck length ratio with respect to the biostructure. Figure 4A illustrates three slices along an isolated anatomical feature, such as the aorta, when the associated lesion is an aneurysm, with respect to which the cross-sectional area has been calculated and displayed on a 3D surface mesh model of the aorta. Figure 4B illustrates, as an example, how slices can be obtained along the axes of a complex structure for the purpose of calculating their cross-sectional area.

[0082] Referring again to Figure 3, in step 310, the 3D volume between each adjacent slice may be extrapolated based on the cross-sectional area of ​​the isolated anatomical feature in the adjacent slice, thereby the overall volume of the isolated anatomical structure may be determined based on the extrapolated 3D volumes, for example, by calculating the sum of the extrapolated 3D volumes. Alternatively, the volume of an automatically segmented label for an isolated anatomical feature, such as a specific part of the left atrial appendage of the heart, may be calculated based on the number of voxels in the semantically labeled portion of the biostructure, so that the volume can be displayed to the user for assessment.

[0083] Referring here to Figure 5, an exemplary method 500 for generating central line measurements of patient-specific anatomical features is provided. As described above with respect to step 210 of method 200 for multi-scheme analysis of patient-specific anatomical features from medical images in Figure 2, physiological information, such as central line measurements of patient-specific anatomical features associated with selected lesions, can be generated from the generated 3D surface mesh model. For example, in step 502, specific anatomical features can be separated from patient-specific anatomical features based on selected lesions in the medical image, as described above, so that a 3D surface mesh model of the separated anatomical features can be extracted and recorded from the 3D surface mesh model of the patient-specific anatomical features.

[0084] In step 504, the start and end points of the isolated anatomical feature are determined at the opposite ends of the isolated anatomical feature, such that a direction vector extending from the start to the end point can be determined. For example, the start and end points may be determined via a machine learning algorithm that searches an anatomical knowledge data set to derive the start and end points of the isolated anatomical feature. The start and end points may also be close to the bounding box of the 3D surface mesh model and lie on a common plane. Furthermore, the initial direction of travel may be determined to be consistent with the direction vector extending from the start to the end point.

[0085] In step 506, a predetermined step size may be determined so that cross-sections can be established along the axes of the isolated anatomical features at regular intervals defined by the predetermined step size. The cross-section at each interval may be perpendicular to the direction of progression associated with the interval. For example, the initial cross-section may be perpendicular to the initial direction of progression based on a direction vector extending from the start point to the end point. Furthermore, multiple rays (e.g., three rays) may be projected along the cross-section at each interval in multiple predetermined directions, perpendicular to the direction of progression and radially outward toward the periphery of the isolated anatomical features, so that the positions of the intersections between the projected rays and the 3D surface mesh model can be determined. For example, as shown in Figure 6, a first set of three rays projected in the direction of propagation from the starting point SP to the ending point EP may intersect at points 602a, 602b, and 602c with isolated anatomical features, such as a 3D surface mesh model of a blood vessel V. In step 506, if it is determined that the point from which the rays are projected is outside the 3D surface mesh model, the point may be moved into the 3D surface mesh model.

[0086] In step 508, the center point of the isolated anatomical feature in the cross-section at each interval, e.g., CP1, may be determined, for example, by triangulating the distances between the intersections of the isolated anatomical features, e.g., points 602a, 602b, and 602c. In step 510, a new direction of progression may be determined at each interval based on direction vectors extending from the previous center point and the current center point of the previous interval. For example, in Figure 6, the new direction of progression at the first interval may be consistent with the direction vector extending from the starting point SP to the center point CP1. If the isolated anatomical feature is a branched vessel, steps 506-510 may be repeated through both branches of the vessel, thereby generating a centerline for each branch of the 3D surface mesh model of the vessel.

[0087] Method 500 may repeat steps 506-510 until an endpoint EP is reached. For example, as shown in Figure 6, in a second interval, three rays may be projected along a cross-section perpendicular to the direction of travel defined by a direction vector extending from a starting point SP to a center point CP1. The distances between the intersections 604a, 604b, and 604c of the projected rays and the 3D surface mesh model may be triangulated to determine the center point CP2 in the second interval. The previous direction of travel may then be adjusted to a new direction of travel defined by a direction vector extending from center point CP1 to center point CP2. Similarly, in a third interval, three rays may be projected along a cross-section perpendicular to the direction of travel defined by a direction vector extending from center point CP1 to center point CP2. The distances between the intersection points 606a, 606b, and 606c of the projected rays and the 3D surface mesh model are triangulated to determine the center point CP3 at a third interval. The previous direction of propagation can then be adjusted to a new direction of propagation defined by a direction vector extending from center point CP2 to center point CP3. As described above, steps 510–512 are repeated until the endpoint EP is reached, thereby determining a series of center points CP along the axis of the isolated anatomical feature, as shown in Figure 7A. Thus, as described above, the points onto which the rays are projected are outside the 3D surface mesh model beyond the endpoint EP, thereby the points cannot be returned to the 3D surface mesh model, and will thus indicate the ends of the centerlines of the isolated anatomical feature.

[0088] In step 512, the centerline of the isolated anatomical feature can be determined based on the totality of the center points, e.g., CP1, CP2, CP3...CPn. For example, the centerline may be a line drawn through all of the calculated center points of the isolated anatomical feature, as shown in Figure 6. Figure 7B illustrates the centerline CL of the isolated anatomical feature as a line drawn through all of the center points CP in Figure 7A. Therefore, the total length of the centerline CL of the isolated anatomical feature can be determined, as shown in Figure 7C. Figure 7D illustrates a 3D surface mesh model of the isolated anatomical feature from Figures 7A-7C, crossing the original medical image.

[0089] Referring here to Figure 8, Method 500 may be used to determine the centerline of an extensive network of patient-specific anatomical features. For example, Figure 8A illustrates the start and end points determined for a 3D surface mesh model of isolated anatomical features. Figure 8B illustrates the centerline CL determined with respect to isolated anatomical features mapped to the original medical image. Figure 8C illustrates the centerline CL for an anatomical feature with multiple vessels, and Figure 8D illustrates the centerline CL for an anatomical feature with an extensive network of vessels.

[0090] Referring here to Figure 9, an exemplary method 900 for generating surface length measurements of patient-specific anatomical features is provided. As described above with respect to step 210 of method 200 for multi-scheme analysis of patient-specific anatomical features from medical images in Figure 2, physiological information, e.g., patient-specific anatomical features associated with selected lesions, e.g., surface length measurements, can be generated from a generated 3D surface mesh model. For example, in step 902, a specific anatomical feature may be separated from patient-specific anatomical features based on selected lesions in the medical image, as described above, thereby a 3D surface mesh model of the separated anatomical features can be extracted and recorded from the 3D surface mesh model of the patient-specific anatomical features.

[0091] In step 904, the centerline of the isolated anatomical feature may be determined, for example, via method 500 as described above. In step 906, the start and end points of the isolated anatomical feature may be determined, for example, at the opposite ends of the isolated anatomical feature. In step 908, a predetermined step size may be determined such that the cross-sections can be established at regular intervals defined by the predetermined step size along the axis of the isolated anatomical feature. As shown in Figure 10, the cross-sections of isolated anatomical features at each interval, for example, of a blood vessel V (e.g., P1, P2), may be perpendicular to the direction of progression associated with the interval (e.g., the direction of progression of the centerline in the interval as described above), and at each interval and point, along the direction vector DV extending from the start point SP to the end point EP, may include center points (e.g., CP1, CP2) along the centerline CL.

[0092] In step 910, rays (e.g., rays R1, R2) can be projected radially outward toward the 3D surface mesh model from their respective center points (e.g., center points CP1, CP2) at each interval along each section (e.g., section P1, P2), such that the positions of the intersections (e.g., intersections D1, D2) between the rays and the 3D surface mesh model are recorded. Step 10 can be repeated at each predetermined interval to determine a series of intersections along the surface topology of the 3D surface mesh model. In step 912, the total length of a line extending across the surface of the 3D surface mesh model of isolated anatomical features can be calculated based on the determined intersections, as defined by the intersections determined in step 910. Figure 11 illustrates a surface line SL extending across the surface topology of the 3D surface mesh model of isolated anatomical features.

[0093] For example, with respect to cardiac image segmentation, once automated segmentation is complete, a 3D surface mesh model of the blood vessels surrounding the heart will be created. This 3D data can then be automatically analyzed to assess specific lengths of cardiac landmarks, which may include, but are not limited to, the atria, ventricles, aorta, vena cava, mitral valve, pulmonary valve, aortic valve, tricuspid valve, myocardium, coronary arteries, and left atrial appendage.

[0094] Referring here to Figure 12, an exemplary method 1200 for generating a heatmap of patient-specific anatomical features based on volume is provided. As described above, the cross-sectional areas of isolated anatomical features at predetermined intervals along the axes of isolated anatomical features can be determined so that a heatmap of the 3D surface mesh model can be generated based on the cross-sectional areas of the 3D surface mesh model along the axes of isolated anatomical features. For example, in step 1202, a particular anatomical feature can be isolated from patient-specific anatomical features based on a selected lesion in the medical image, as described above, so that a 3D surface mesh model of the isolated anatomical features can be extracted and recorded from the 3D surface mesh model of patient-specific anatomical features. In step 1204, the start and end points of the isolated anatomical feature can be determined so that the initial direction of progression is consistent with a direction vector extending from the start to the end point. In step 1206, the centerline of the isolated anatomical feature can be determined, for example, via method 500 described above.

[0095] In step 1208, a predetermined step size may be determined such that slices can be obtained at regular intervals defined by the predetermined step size along the midline of the isolated anatomical feature. Thus, slices of the isolated anatomical feature can be obtained at each interval perpendicular to the direction of progression along the midline. In step 1210, a standard calculation function may be used to calculate the cross-sectional area in each slice of the isolated anatomical feature such that it is defined by the periphery of the isolated anatomical feature. In step 1210, a heatmap may be generated based on the cross-sectional area in each slice of the 3D surface mesh model, thereby visually showing the change in volume throughout the isolated anatomical feature, as shown in Figures 13A and 13B.

[0096] Referring here to Figure 14, an exemplary method 1400 is provided for generating a heatmap of patient-specific anatomical features based on the degree of meandering. As described above, the direction of progression at predetermined intervals of the centerlines of isolated anatomical features can be determined so that a heatmap of the 3D surface mesh model can be generated based on the magnitude of the change in direction of progression along the axes of the isolated anatomical features. For example, in step 1402, a particular anatomical feature can be isolated from patient-specific anatomical features based on a selected lesion in the medical image, as described above, so that a 3D surface mesh model of the isolated anatomical features can be extracted and recorded from the 3D surface mesh model of patient-specific anatomical features. In step 1404, the start and end points of the isolated anatomical features can be determined so that the initial direction of progression is consistent with a direction vector extending from the start to the end point. In step 1406, the centerlines of the isolated anatomical features can be determined, for example, via method 500 described above.

[0097] In step 1408, the direction of progression of the midline of the isolated anatomical features at predetermined intervals may be determined, for example, based on direction vectors extending between adjacent midpoints along the midline, as described above. In step 1410, the magnitude of the change between the directions of progression of adjacent intervals may be determined. For example, the magnitude of the change may be calculated using the direction vectors associated with each direction of progression in each interval. In step 1412, a heatmap is generated based on the magnitude of the change between the directions of progression of adjacent intervals along the axes of the 3D surface mesh model, thereby visually representing the degree of meandering of the isolated anatomical features, as shown in Figure 15. Thus, the magnitude of the change, for example, the angle change output from the analysis, may be cross-referenced with an existing knowledge database of known classifications of angle deviations and displayed to the user. The meandering value is described as the total change in the angle of the vessel and may be scored, for example, as a 760-degree rotation score.

[0098] Referring here to Figure 16, an exemplary method 1600 is provided for generating a 3D surface mesh model of patient-specific anatomical features using identified markers. As described above with respect to Figure 2, a medical image, such as that shown in 1702 of Figure 17A, can be automatically processed so that a 3D surface mesh model of classified patient-specific anatomical features within the medical image can be generated, and patient-specific anatomical features, such as those shown in 1704 of Figure 17A, can be identified. Method 1600 further identifies corresponding markers (e.g., bone notches or heart valves) of the patient-specific anatomical features so that the markers can be depicted in the 3D surface mesh model. For example, prior to the generation of a 3D surface mesh model based on classified patient-specific anatomical features, in step 1602, information indicating a specific anatomical feature can be separated from data representing patient-specific anatomical features based on a selected lesion in the medical image, as shown in 1706 (biological structure boundary) of Figure 17A.

[0099] In step 1604, the features of the isolated anatomical features are analyzed using an anatomical feature data set, and one or more markers of the isolated anatomical features associated with the selected lesion may be identified. For example, the anatomical feature data set may include knowledge of anatomical markers, e.g., existing semantically labeled anatomical feature data sets associated with various patient-specific anatomical features, thereby the markers may be identified and individually labeled by establishing a link between the classified isolated anatomical features and the anatomical feature data set. In step 1606, the identified labeled markers may be associated with pixels of the original medical image, as shown in 1708 of Figure 17A. In step 1608, a 3D surface mesh model of the isolated anatomical features may be generated to depict the identified markers mapped to pixels of the medical image associated with the identified markers, as shown in 1710 of Figure 17A.

[0100] Identified anatomical landmarks are significant points within a patient's biostructure that have morphological or functional significance, such as orientation and insertion points relative to other anatomical features. Identified landmarks can help surgeons ensure that landmarks correspond to specific parts of biostructures and that their proper function and orientation are ensured. Identified landmarks can be further utilized in clinical practice as markers on biostructures, for example, as initial references for anatomical guide fixation and trajectory planning, thereby facilitating the diagnosis and / or treatment of patients. For example, specific identified anatomical landmarks for each bone can be automatically detected so that guides can be generated for bone cutting and drilling. Thus, identified anatomical landmarks can be used as inputs for clinical functions that have significant benefits. For example, Figure 17B illustrates the following identified landmarks mapped to isolated anatomical features (e.g., the scapula for shoulder replacement): (A) fovea, (B) triquetrum, (C) inferior angle, and (D) center of the scapular spine. Therefore, identified landmarks can serve as references to provide guidance for cutting planes within bone, drilling trajectories, and fixing devices within bone.

[0101] Referring here to Figure 18, an exemplary method 1800 for identifying medical devices and treatment options for a lesion is provided. For example, in step 1802, a specific anatomical feature may be separated from a patient-specific anatomical feature based on a selected lesion in a medical image, as described above, so that a 3D surface mesh model of the separated anatomical feature may be extracted and recorded from a 3D surface mesh model of the patient-specific anatomical feature, as shown in Figure 19A. In step 1804, the physiological parameters of the separated anatomical feature may be analyzed, as described above, to determine measurements such as volume, centerline, surface length, cross-sectional area, diameter, and density.

[0102] Based on the physiological parameters of isolated anatomical features and patient background data associated with medical images, step 1806 may identify one or more medical devices and / or treatment options from a medical device database having knowledge of various medical devices, including their functions and specifications, and / or a surgical implementation database having knowledge of lesion-specific treatment options. For example, the physiological parameters of isolated anatomical features may indicate the size of a selected lesion so that a medical device of a particular size, known to be used to treat the selected lesion, can be identified for use in treating the lesion. The identified medical device may further be selected from internal inventory, e.g., available medical devices, or may be provided by or supplied by a particular hospital. The knowledge data sets described herein may further include knowledge of combinations of biostructures and non-organic materials (e.g., polymers, metals, and ceramics) so that non-organic materials can also be automatically segmented. In addition, the knowledge data sets may include knowledge of medical devices (e.g., knowledge of existing implants for correcting bone lesions) which can be used as input for creating patient-specific guides. For example, known dimensions and variability of a device can be used as input in the automated design of the device. In step 1808, the identified medical devices and / or treatment options may be displayed to the user so that the user can make informed decisions regarding preoperative planning and treatment, as shown in Figures 19B and 19C.

[0103] The ability to provide automated segmentation opens up several beneficial lesion-specific applications. For example, several specific lesions / treatments that require higher-level volume 3D models (virtual or physical) are listed in Table 1 below. [Table 1-1] [Table 1-2]

[0104] Referring here to Figure 20, an exemplary method 2000 for detecting and classifying fractures of patient-specific anatomical features is provided. As described above with respect to Figure 2, a medical image, such as that shown in Figure 21A, is automatically processed so that a 3D surface mesh model of the classified patient-specific anatomical features in the medical image can be generated and the patient-specific anatomical features can be identified. Method 1600 further detects / identifies the corresponding fractures of the patient-specific anatomical features in the bone (e.g., fractures in the tibia, fibula, or medial malleolus, etc.) so that the features can be depicted in the 3D surface mesh model. For example, prior to the generation of a 3D surface mesh model based on the patient-specific anatomical features classified in step 2002, information indicating a specific anatomical feature can be separated from data representing the patient-specific anatomical feature based on a selected lesion in the medical image, as shown in Figures 21B and 21C.

[0105] In step 2004, the features of isolated anatomical features are analyzed using an anatomical feature data set to identify one or more markers (e.g., notches) of isolated anatomical features associated with a selected lesion. As described above, the anatomical feature data set may include knowledge of anatomical markers associated with various patient-specific anatomical features so that the markers can be identified and individually labeled by establishing links between the classified isolated anatomical features and the anatomical feature data set. In step 2006, the features of the identified markers are analyzed using a reference fracture database to identify one or more fractures of the identified markers associated with a selected lesion of the isolated anatomical features. The reference fracture database may include knowledge of various fractures (e.g., existing semantically labeled reference fracture data sets associated with various patient-specific anatomical features) so that fractures can be identified and individually labeled by establishing links between the classified isolated anatomical features and the anatomical feature data set. In step 2008, a 3D surface mesh model of the isolated anatomical features may be generated to depict the identified landmarks and detected fractures F, as shown in Figure 21D. Furthermore, in step 2010, the 3D surface mesh model may be matched against a reference fracture database to classify the fracture type.

[0106] Referring here to Figure 22, an exemplary method 2200 for tracking the temporal progression of a lesion is provided. For example, in step 2202, specific anatomical features may be separated from patient-specific anatomical features based on a selected lesion in a medical image, as described above, so that a 3D surface mesh model of the separated anatomical features can be extracted and recorded from a 3D surface mesh model of patient-specific anatomical features. In step 2204, the physiological parameters of the separated anatomical features may be analyzed, as described above, to determine measurements such as volume, centerline, surface length, cross-sectional area, diameter, and density.

[0107] For example, once automated segmentation is complete, a 3D surface mesh model of the aneurysm and vascular biostructure can be generated. This 3D data can then be automatically analyzed to assess specific lengths relating to the aneurysm morphological structure, which may include, but are not limited to, measurements of the aneurysm neck, measurements of the aneurysm diameter at maximum distance, and measurements of the center points of the superior and inferior aneurysm necks.

[0108] In step 2206, the physiological parameters to be analyzed for isolated anatomical features may be time-stamped and recorded so that a time-series record of the physiological parameters for a particular patient exists over time. In step 2208, the temporal changes between the recorded / time-stamped physiological parameters may be calculated, for example, to indicate the progression and prognosis of a selected lesion. For example, Figures 23A-23F illustrate the temporal growth of various aneurysms, where growth leads to eventual rupture. Optionally, in step 2210, a heatmap may be generated to visually depict the temporal changes between the recorded / time-stamped physiological parameters, as shown in Figures 24A-24F.

[0109] Referring here to Figure 25, an exemplary method 2500 for semantic volume rendering is provided. A single medical image 2602 of a stack of medical images 2604 is shown in Figure 26. Volume rendering is an important solution globally adopted by healthcare professionals to visualize sets of medical imaging data in 3D space. They work by mapping pixel properties such as specific color, intensity, or opacity to specific voxels in a 3D scene. There are drawbacks associated with this imaging method, where overlapping structures and deep structures are not easily visualized in detail. Therefore, to overcome these drawbacks, method 2500 generates a 3D surface mesh model of distinct anatomical features so that the physiological parameters of distinct anatomical features can be analyzed.

[0110] For example, the results of automated image segmentation may take the form of a series of binary pixel arrays contained in a medical image, such as a DICOM file. When assembled into a volume, the binary pixel array can be used to mask areas of the source pixel volume that are not related to the identified biological structure. The remaining Hounsfield value volume can then be rendered using standard volume rendering techniques and color transfer functions, such that the pixel intensity can be determined based on the Hounsfield values. Furthermore, the length of anatomical features (e.g., blood vessels) can be calculated based on the output from the automated segmentation algorithm and subsequent 3D reconstruction. The data extracted from the 3D reconstruction can then be automatically analyzed and output the length from one specific anatomical landmark or anomaly to another (e.g., the length from the aortic arch to the thrombus in the case of a stroke). In the case of blood vessels, the measurement can be calculated by creating a center point on the cross-section of the blood vessel and an extrapolated center point through the blood vessel, joining the center points, and creating a centerline of the biological structure. This centerline can then be automatically measured and output to the user as a length value.

[0111] For example, in step 2502, classified patient-specific anatomical features generated using the segmentation algorithm described above are depicted in binary labels (e.g., bone / non-bone, vascular / non-vascular, organ / non-organ, etc.). In step 2504, the binary labels are divided into distinct anatomical features, e.g., cardiac myocardium, aorta, coronary arteries, etc. In step 2506, the distinct anatomical features are mapped to the original medical image such that only the original grayscale values ​​or Hounsfield units for the distinct anatomical features are shown in the medical image, as shown in 2606 and 2608 of Figures 26 and 27B, and the background is removed from the medical image, as shown in 2610 and 2612 of Figures 26 and 27C, leaving only the distinct anatomical features depicted in the original grayscale values ​​or Hounsfield units visible.

[0112] In step 2508, a 3D surface mesh model of a separate anatomical feature may be generated. The 3D surface mesh model may define the surface of the separate anatomical feature, as shown in Figure 26, 2614. In addition, or alternatively, a specific color of transparency values ​​may be mapped to the labeled 3D surface mesh model, as shown in Figures 26 and 27, 2616, to generate volume rendering. For example, a color map of pixel intensity may be directly mapped to the 3D voxel intensity within the segmentation only, enabling the visualization of specific volumes of isolated anatomical features. Voxels may be automatically assigned a specific color depending on the intensity of the original image, which may indicate normal blood flow or absence of blood flow. The ability to color specific areas of interest, such as blood clots, destruction, or biostructures, allows for further insight into specific lesions in the region.

[0113] As shown in Figure 27D, 3D volume rendering can indicate the presence of blood clots / occlusions. This data can then be rendered on an end-user application so that the 3D volume rendering can be rotated or otherwise manipulated and viewed. This data can also be used, for example, to indicate to the user whether calcification is present from clusters of high-intensity pixels, and further, to provide a calcification "score" by indicating the percentage of blood clots or occlusions representing calcified structures. For example, occlusion / calcification predictions can be made and applied as a mask on the original medical image, thereby removing the background portion of the medical image, as shown in 2802 of Figure 28A. Thus, 3D surface mesh models that take into account the pixel intensity of various materials can be generated, as shown in 2804, 2806, and 2808 of Figure 28A. As shown in Figure 28B, the size of the occlusion O depicted in the vascular V of the 3D volume rendering can be measured, for example, to assist in the diagnosis and treatment of stroke patients.

[0114] 3D volume rendering can visualize specific features by referencing anatomical features that are set by the user or automatically derived and depicted in volume renderings, such as vascular structures, coronary arteries, and blood clots within nerve vessels, thereby indicating potential strokes. Therefore, medical images can be automatically segmented and reconstructed by creating 3D representations of both the vessels and associated occlusions using machine learning from semantically labeled 3D anatomical knowledge data sets that can be easily viewed on mobile devices or similar platforms, for example, utilizing CTA / XA / NM vascular imaging for patients.

[0115] When a 3D surface mesh model is generated from automated segmentation, it may be possible to generate several measurements of biological structures or lesions within a medical scan. Furthermore, scaling information, along with reference points, allows for the placement of patient-specific anatomical features within the physical scene. At its simplest level, physical measurements of the mesh, any submesh, or otherwise demarcated areas within the physical scene may be generated, which may include mesh length, spread, height, angle, curvature, meandering, etc. Assuming a filled structure, volume, surface area, and diameter measurements may also be performed.

[0116] The derived properties of the material to be segmented can also be measured. At a basic level, these may include the thickness of the material (blood vessel or bone) and known deviations from normal values ​​(patient or general), which may, for example, enable the generation of predictions about the pressure likely to be required to break the material, or simply provide visualization of thickness and stress lines. Visualization of any of the measurements described above would be of great value, as any more information available to the surgeon would be useful in determining the best course of action for treatment and would provide the ability to give accurate analysis of the diagnosis. This may be achieved through a simple overlay of derived variables onto the mesh, or by providing data for additional analysis of input / desired attributes.

[0117] As described above, aside from determining the structure of patient-specific anatomical features, the extracted polygonal model can further provide a useful basis for determining numerous useful measurements that would otherwise be difficult to ascertain from volume pixel data alone, such as bone and vascular dimensions, angle and meandering differences, and relative scale, density, etc. Typically, determining these measurements would require careful manual assessment of the mesh to identify areas of interest and meaningful reference points. However, the exploratory geometric algorithms described herein offer a reliable automated alternative. For example, the following pseudocode outlines how vascular length, diameter, and curvature information can be automatically collected without human intervention. [ka]

[0118] Referring here to Figure 29, an exemplary method 2900 for analyzing the physiological parameters of distinct anatomical features is provided. Some of the steps of method 2900 can be further detailed by referring to Figures 30A–30E, which depict 2D examples of cross-sections of vessels with branching pathways. Figure 30A illustrates a branched vessel V. In step 2902, planes P1, P2, and P3 can be constructed at the inlet points of vessel V, defined by the volume boundaries of vessel V, as shown in Figure 30B. In step 2904, the center points C1, C2, and C3 of the inlet planes P1, P2, and P3 can be calculated, as shown in Figure 30C, respectively. As shown in Figure 30C, multiple rays can be projected from center point C3 into the structure of vessel V to determine the longest unobstructed path within vessel V. Due to the branching pathways of vessel V, there are two peak points PP1 and PP2, depicted in Figure 30C. This can be determined by assessing the number of inflection points in the graph of distance values. At this point, since it is determined that there are many paths ahead in the algorithm, each branch can be assessed individually by separating from the control flow.

[0119] In step 2906, the entire structure of the vessel V is advanced through the vessel V until rays projected along lines L1 and L2 at each point intersect with the entrance surfaces P2 and P3, respectively, as shown in Figure 30D, resulting in a series of vertices that create each of the paths through the vessel V. In step 2908, the best-fit spline can be constructed along lines L1 and L2 through the vertices, as shown in Figure 30E, thereby obtaining diameter measurements at each point along lines L1 and L2, which can provide a complete representation of the vessel V, and the inclination / meandering, diameter, internal volume, etc. of the vessel V can be determined.

[0120] Furthermore, from the pseudocode described above, the presence of lesions such as aneurysms will result in search points that get stuck in the loop. Each time a point on the measurement line begins to repeatedly change direction, the algorithm can infer that it has exited the search loop and entered the aneurysm. Therefore, physiological measurements of an aneurysm can be determined, for example, by determining points around the entrance to the aneurysm, constructing an entrance plane relative to the aneurysm, determining the center point of the entrance plane, projecting a ray into the aneurysm structure, determining the furthest point, and once the maximum distance is determined, initiating a check of the vertical distance by constructing a line between the entrance plane and the furthest point and projecting a ray.

[0121] The results of segmentation can be quantified, particularly in the context of oncology, by, for example, measuring the density of the segmented area, identifying its proximity to other parts of the biological structure, and identifying and drawing its boundaries. Once a region is identified and drawn within a physical context, a description of the region can be made in relation to other structures within that context. For example, drawing tumor boundaries and understanding their distance from important structures within the anatomical periphery would be useful for an oncologist. Furthermore, the density of a given structure may provide clinically relevant information, such as, in the case of oncology, insight into intratumor hypoxia, or, in the case of blood clots, insight into how the clot may be treated.

[0122] The ability to measure the density and thickness of anatomical regions would enable the ability to provide guidance, for example, on screw selection in trauma applications or catheter diameter in vascular applications. Furthermore, the ability to measure diameter along anatomical features would allow diameter measurements to be cross-referenced with medical device databases, enabling surgeons to indicate the best-sized device for that patient.

[0123] The machine learning-based algorithms described herein may be trained and predicted on the axial axis, which is typically the axis along which medical scans are performed. Modifications to the machine learning-based algorithms may involve changing the prediction function, and other modifications may involve changing the training and prediction function. For example, a modification to the machine learning-based algorithm may involve making predictions on all three axes and then merging the results. This approach would work best when the voxels are isotropic, as in the case of rimasys data. Merging predictions may follow several different strategies, such as taking the average (mean) of the three results for a given pixel / voxel, or a more complex solution such as finding a weighted average of the axial slices plus others. Alternatively, it may be possible to switch to a different primary axis, for example, from the axial axis to the sagittal axis.

[0124] Training algorithms on all three axes allows for the utilization of additional information from different axes. Thus, axial inference models, sagittal inference models, and coronal models can be trained. As described above, the results of all three predictions can be combined using a simple merge strategy. However, preferably, the output layers of any one of the three models can be combined in a larger network, or an ensemble model can be created to combine their results.

[0125] As described in U.S. Patent Application Publication 2021 / 0335041, the algorithm could inherently function in 3D, which could be very expensive in terms of memory allocation. One alternative approach to alleviate this limitation would be to consider cubes at once, rather than slices. The advantage of this approach is that, instead of considering large, thick slabs, the algorithm may be able to consider a more relevant and immediate context during training, as it is trained on a volume of small cubes that slide across the entire volume.

[0126] The sandwich approach described in U.S. Patent Application Publication 2021 / 0335041 can be extended to incorporate a larger number of slices and more explicitly incorporate pixels from surrounding slices into the model. For example, instead of using additional channels within the image, multiple channels of most image formats, e.g., three channels, can be leveraged to achieve this compression. By embedding surrounding images within the complete image, the number of surrounding images in a scan can be increased overall. As the size of the GPU increases, the number of surrounding images in a scan can also increase. Furthermore,

[0127] The algorithm can implement a version of D-Unet that takes 3D context information into account (via a 3D convolutional kernel), increasing the amount of slices the model analyzes at once and providing the algorithm with much more spatial context. This architecture has been upgraded along with improvements to the loss function, and access to more data is leading to increasingly better segmentation models.

[0128] Furthermore, the methods described herein can further utilize the Euclidean distance-weighted approach to influence the loss component in the machine learning model training process. This approach helps guide the learning process and focus on areas of greater importance. For example, in orthopedic segmentation, the errors that are most difficult to detect / identify and repair are small connections between bones that are very close to each other, while small holes inside the bone are simpler to correct. Figure 31 illustrates the weighted mask generated using the Euclidean distance-weighted approach and its effect on the loss function, e.g., classification cross-entropy.

[0129] A multi-scheme approach to ground truth data sets for training is provided. Specifically, there are many different segmentation marking schemes that can be used to adapt training marks depending on the goals of the model to be trained. For example, since defining the medial material of trauma bone can be very difficult, they are generally segmented as hollow, and therefore, predictions from trauma models trained on hollow bone markings are much easier to work, as shown in Table 2 below. [Table 2]

[0130] Figure 32 illustrates various bone segmentation schemes in medical images using a multi-scheme approach to ground truth data for training purposes, as described above. Similarly, Table 3 illustrates cardiac segmentation labeling schemes used in conjunction with a multi-scheme approach to ground truth data. [Table 3]

[0131] Figure 33 illustrates various segments of myocardium in medical images of ground truth data for training purposes.

[0132] These same techniques for adapting labeling schemes can be used to distinguish between normal and pathological tissue, or in some cases, to delineate tissue absence, which would enable semantic segmentation of lesions as areas of interest and further allow lesion-specific workflows to be automatically initiated. Furthermore, a multi-scheme approach using multiple labels to distinguish between biological structures and lesions can be used to semantically label each anatomical feature of the human body.Examples of various scheme markers are, but are not limited to, the nose, lacrimal gland, inferior nasal concha, maxilla, zygomatic bone, temple, palate, vertebral column, malleus, incinus, stapes, frontal bone, ethmoid bone, vomer, sphenoid bone, mandible, occipital bone, rib 1, rib 2, rib 3, rib 4, rib 5, rib 6, rib 7, rib 8 (pseudorib), rib 9 (pseudorib), rib 10 (pseudorib), rib 11 (floating rib), rib 12 (floating rib), hyoid bone, sternum, cervical 1 (atlas), C2 (axis), C3, C4, C5, C6, C7, thoracic 1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T1 1, T12, Lumbar vertebra 1, L2, L3, L4, L5, Sacrum, Coccyx, Scapula, Clavicle, Humerus, Radius, Ulna, Scapula, Lunate, Triquetrum, Pisiform, Hamate, Capitate, Trapezoid, Trapezoid, Metacarpal 1, Proximal phalanx 1, Distal phalanx 1, Metacarpal 2, Proximal phalanx 2, Middle phalanx 2, Distal phalanx 2, Metacarpal 3, Proximal phalanx 3, Middle phalanx 3, Distal phalanx 3, Metacarpal 4, Proximal phalanx 4, Middle phalanx 4, Distal phalanx 4, Metacarpal 5, Proximal phalanx 5, Middle phalanx 5, Distal phalanx 5, Hip joint (Ilium, Ischium, Pubis), Femur, Patella, Tibia, Fibula, Talus, Calcaneus, Scaphoid, Medial cuneiform, Intermediate cuneiform, Lateral Lateral cuneiform bones, cuboid bone, metatarsal 1, proximal phalanx 1, distal phalanx 1, metatarsal 2, proximal phalanx 2, middle phalanx 2, distal phalanx 2, metatarsal 3, proximal phalanx 3, middle phalanx 3, distal phalanx 3, metatarsal 4, proximal phalanx 4, middle phalanx 4, distal phalanx 4, metatarsal 5, proximal phalanx 5, middle phalanx 5, distal phalanx 5, ring of Willis, anterior cerebral artery, middle cerebral artery, posterior cerebral artery, lenticular striatal artery, brachiocephalic artery, right common carotid artery, right subclavian artery, vertebral artery, basilar artery, posterior cerebral artery, posterior cerebral artery, posterior communicating artery, left common carotid artery, internal carotid artery (ICA), external carotid artery (ECA), left subclavian artery, right subclavian artery This may include the internal thoracic artery, thyrocervical artery, costocervical artery, left subclavian artery, aorta, vena cava, axilla, axillary artery, brachial artery, radial artery, ulnar artery, descending aorta, thoracic aorta, abdominal aorta, internal iliac artery, external iliac artery, femoral artery, popliteal artery, anterior tibial artery, dorsalis pedis artery, posterior tibial artery, tricuspid valve, pulmonary valve, mitral valve, aortic valve, right ventricle, left ventricle, right atrium, left atrium, liver, kidney, spleen, intestine, prostate, cerebrum, cerebral axis, cerebellum, pons, medulla, spinal cord, frontal lobe, parietal lobe, occipital lobe, temporal lobe, right coronary artery, left major coronary artery, left anterior descending branch, and left circumflex branch.

[0133] A hybrid data labeling method for reinforcement learning is provided. For most machine learning models, creating a large corpus of data for training is essential. Regarding segmentation algorithms for labeling DICOMS as described herein, the ability to create large amounts of data for a robust algorithm is limited by the resources of skilled engineers or imaging specialists. By leveraging the initial results of the segmentation algorithm, the method described herein can accelerate the time required to create large datasets. For example: Time required to segment a single image (unautomated) = 10 seconds; Assumptions for a robust algorithm: 100,000 labeled images; The sequential segmentation of 100,000 images will take approximately 278 hours;

[0134] A theoretical study example where the model was trained four times and the algorithmic training was linear: 0-25,000 sheets - approximately 69 hours - training; 25,001-50,000 (25% completed by algorithm) 52 hours - retraining; 50,001-75,000 (50% completed by algorithm) 35 hours - retraining; 75,001-100,000 (75% completed by algorithm) 17 hours 100,000 images segmented using a hybrid of algorithms and techniques - 173 hours;

[0135] The simplified example above demonstrates that a segmentation algorithm, with the help of retraining, could achieve the desired level of automation much faster. Furthermore, it could be further developed by retraining the algorithm after each data set has been added to the training set. This can be achieved by using cloud infrastructure and event-driven serverless computing platforms such as AWS Lambdas. Presenting the user with the updated set of labels after each retraining could dramatically reduce the time required to generate large amounts of data.

[0136] Furthermore, most medical image segmentation applications require a very high level of accuracy, and therefore, medical images can be used at their original full resolution. However, when there is a specific need to consider all or most of a 3D scan to detect a lesion, such as an aneurysm, most 2D-based approaches will not suffice. Moreover, due to limitations in current hardware or exorbitant costs, 3D approaches may not be applicable to full-resolution scanning.

[0137] Therefore, the method described herein can downsample the volume of examination and identify important features by segmenting the vascular system within a CT scan, e.g., a nerve CT scan, using a D-Unet-based architecture. This architecture considers small stacks of 2D images, e.g., four lower slices and four upper slices, thereby providing some small 3D contextual information. In the case of aneurysm detection, current approaches may not be sufficient to distinguish between aneurysms and healthy blood vessels when considering only a few 2D images at once, and may not be sufficient to achieve the context required to correctly identify aneurysms. This is mainly because the texture and general appearance of an aneurysm are indistinguishable from other vascular systems when considered alone, e.g., within a few 2D images.

[0138] The ability to automatically identify, potentially locate, and measure aneurysms, blood clots, and occlusions could revolutionize neurosurgical procedures and save lives. For example, the method described herein may employ a more advanced approach that considers a holistic scan from a 3D perspective to distinguish these abnormalities from the rest of the vascular system. Thus, the method described herein may implement a two-step approach, where the first step identifies the vascular system within a stack of images using a full-resolution approach, and then, in the second step, a separate model considers a low-resolution version of the scan in three dimensions. After acquiring a region where an aneurysm is located within the low-resolution volume, the region can be co-located with a high-resolution version so that the aneurysm can be segmented from the general vascular system segmentation. This approach has numerous potentials for other high-resolution 3D volume applications where there is a need to distinguish similarly textured elements that require a much larger context to be correctly identified.

[0139] Image preparation for the purpose of generating models (physical or virtual) using actual medical images requires a certain amount of pre-filtering and refinement to generate accurate models. Therefore, several transformations must be performed on the images to dramatically improve the quality of the final model.

[0140] For example, image interpolation can be quite acceptable because large datasets of existing images can be used to train algorithms. This type of problem is particularly well-suited for adversarial networks. Furthermore, image registration can be important because the number of cases involving multiple scanning modalities is increasing, and there may be a need to register CT->MRI images. For example, images from multiple scanning modalities can be registered by aligning two different datasets together. For instance, if a medical scan of a patient's head is provided, with tumors desired from the MRI scan and bones desired from the CT scan, landmarks can be acquired so that they are visible in both the MRI and CT scans to register pixels and voxels in the same location. Even MRI scans where images are obtained in multiple viewpoints / planes within a single session may require registration because differences between planes can produce quite different views of the patient and highlight completely different aspects of biological structures.

[0141] By focusing specifically on the integration required to enable end-to-end processes rather than on individual processes themselves, the systems and methods described herein focus on how to integrate data upstream and downstream of a platform.

[0142] This field may involve all integrations downstream, such as electronic medical / health records. Furthermore, information from EMR (potentially for later correlation with outcomes (see prognosis classification)) may be collated, which would include any upstream integration with transport companies or printing offices, etc. A key value in this field is demonstrating the digital thread of the generation of all models, from the initial idea of ​​data origin to the manufactured objects / virtual objects and beyond, based on the data inflow.

[0143] While various illustrative embodiments of the present invention are described above, it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the invention. The appended claims are intended to cover all such changes and modifications that fall within the true scope of the invention.

Claims

1. A method for multi-scheme analysis of patient-specific anatomical features from medical images, wherein the method is: The server receives the patient's medical image and metadata associated with the medical image that shows the selected lesion. The server automatically processes the medical image using a segmentation algorithm to label the pixels of the medical image and generate a score indicating the likelihood that the pixels are correctly labeled. The server classifies one or more patient-specific anatomical features in the medical image by probabilistically matching the associated groups of the labeled pixels against a set of anatomical knowledge data, wherein the set of anatomical knowledge data includes multiple groups of nodes representing various anatomical features, and the probabilistic matching of the associated groups of the labeled pixels against the set of anatomical knowledge data is performed by identifying the group of nodes in the set of anatomical knowledge data that is most similar to the associated groups of the labeled pixels. The server generates a 3D surface mesh model that defines the surface of one or more classified patient-specific anatomical features, The server extracts information from the 3D surface mesh model based on the selected lesion, The server generates physiological information associated with the selected lesion for the 3D surface mesh model based on the extracted information, wherein the physiological information includes physiological measurements or characteristics related to the diagnosis and / or treatment of the selected lesion, calculated or determined from the extracted information. Methods that include...

2. The method according to claim 1, wherein the information extracted from the 3D surface mesh model comprises a 3D surface mesh model of anatomical features separated from one or more classified patient-specific anatomical features based on the selected lesion.

3. The server generates physiological information associated with the selected lesion for the 3D surface mesh model. Determining the start and end points of the separated anatomical features, Obtaining slices at predetermined intervals along the axis from the starting point to the ending point, Calculate the cross-sectional area of ​​each slice defined by the periphery of the separated anatomical features, Based on the respective cross-sectional areas, the 3D volume between adjacent slices is extrapolated, Based on the extrapolated 3D volume between the adjacent slices, the overall 3D volume of the separated anatomical features is calculated. The method according to claim 2, including the method described in claim 2.

4. The server generates physiological information associated with the selected lesion for the 3D surface mesh model. Determining the starting and ending points of the separated anatomical features, and the direction of progression from the starting point to the ending point, Projecting light rays in at least three directions perpendicular to the direction of travel at predetermined intervals along the axis, and determining the distance between each projected light ray and the intersection point of the 3D surface mesh model, The center point is calculated by triangulating the distance between each projected ray and the intersection point of the 3D surface mesh model at each interval, At each interval, the direction of propagation is adjusted based on the direction vector between adjacent calculated center points, thereby causing ray projection at the predetermined interval to occur in at least three directions perpendicular to the adjusted direction of propagation at each interval. Based on the calculated center point from the starting point to the ending point, the centerline of the separated anatomical features is calculated. The method according to claim 2, including the method described in claim 2.

5. The server generates physiological information associated with the selected lesion for the 3D surface mesh model. Calculating the midline of the separated anatomical features, Determining the start and end points of the separated anatomical features, and the direction vector from the start point to the end point, Based on the direction vector from the starting point to the ending point, a cross-section is established at predetermined intervals along the center line, wherein each cross-section is perpendicular to the direction of travel of the center line at each interval. Projecting light rays onto the cross-section at each interval and determining the position of the intersection point on the 3D surface mesh model from the center line, Based on the determined intersection points at each of the aforementioned intervals, the length across the 3D surface mesh model is calculated. The method according to claim 2, including the method described in claim 2.

6. The server generates physiological information associated with the selected lesion for the 3D surface mesh model. Determining the start and end points of the separated anatomical features, Obtaining slices at predetermined intervals along the axis from the starting point to the ending point, Calculate the cross-sectional area of ​​each slice defined by the periphery of the separated anatomical features, Based on the cross-sectional area of ​​each slice, a heatmap of the separated anatomical features is generated. The method according to claim 2, including the method described in claim 2.

7. The server generates physiological information associated with the selected lesion for the 3D surface mesh model. Determining the start and end points of the separated anatomical features, Calculating the midline of the separated anatomical features, Determining the directional vector between adjacent points along the aforementioned center line, The magnitude of the change in the directional vector between adjacent points along the aforementioned center line is calculated, Based on the magnitude of the change in the directional propagation vector between adjacent points along the center line, a heat map of the separated anatomical features is generated. The method according to claim 2, including the method described in claim 2.

8. The generated physiological information associated with the selected lesion for the 3D surface mesh model includes an associated timestamp. The aforementioned method, The server records the generated physiological information and the associated timestamp. The server calculates the temporal changes between the recorded physiological information indicating the progression of the selected lesion, based on the associated timestamp. The method according to claim 2, further comprising:

9. The method described above is: The server calculates the magnitude of the change over time between the recorded physiological information, The server generates a heatmap of the separated anatomical features based on the magnitude of the temporal changes between the recorded physiological information. The method according to claim 8, further comprising:

10. The server extracts information from the 3D surface mesh model based on the selected lesion. Based on the selected lesion, the anatomical features are separated from the one or more classified patient-specific anatomical features. Using an anatomical feature database, the characteristics of the separated anatomical features are analyzed, and one or more markers of the separated anatomical features are identified. Associating one or more identified markers with the pixels of the medical image, To generate a 3D surface mesh model that defines the surface of the separated anatomical features having the identified markers. The method according to claim 1, including the method described in claim 1.

11. The method described above is: The server identifies a guided trajectory for performing a surgical procedure from a surgical implementation database based on the selected lesion and the one or more identified landmarks. Displaying the aforementioned guidance path to the user The method according to claim 10, further comprising:

12. The method described above is: The aforementioned server receives patient background data, The server identifies one or more medical devices from a medical device database based on the patient background data and the generated physiological information associated with the selected lesion for the 3D surface mesh model. Displaying one or more identified medical devices to the user The method according to claim 1, further comprising:

13. The method described above is: The aforementioned server receives patient background data, The server identifies one or more treatment options from a surgical implementation database based on the patient background data and the generated physiological information associated with the selected lesion for the 3D surface mesh model. Displaying one or more of the identified treatment options to the user The method according to claim 1, further comprising:

14. The server extracts information from the 3D surface mesh model based on the selected lesion. Based on the selected lesion, the anatomical features are separated from the one or more classified patient-specific anatomical features. Using an anatomical feature database, the characteristics of the separated anatomical features are analyzed, and one or more markers of the separated anatomical features are identified. Using a reference fracture database, the characteristics of one or more of the aforementioned landmarks are analyzed to detect fractures of the separated anatomical features, To generate a 3D surface mesh model of the separated anatomical features comprising the one or more identified landmarks and the detected fractures. The method according to claim 1, including the method described in claim 1.

15. The method according to claim 14, further comprising classifying the detected fractures by matching the 3D surface mesh models of the separated anatomical features to the reference fracture database.

16. The method described above is: The server depicts one or more of the classified patient-specific anatomical features in binary labels, The server separates the binary marker into separate anatomical features, The server maps the separate anatomical features to the original grayscale values ​​of the medical image and removes the background within the medical image. It further includes, The method according to claim 1, wherein the generated 3D surface mesh model comprises volume rendering, which is defined by defining the surface of the distinct anatomical features or by mapping specific color or transparency values ​​to one or more classified patient-specific anatomical features.

17. The method according to claim 1, wherein the segmentation algorithm comprises at least one of threshold-based, decision tree, chained decision forest, or neural network methods.

18. The method according to claim 1, wherein the physiological information associated with the selected lesion comprises at least one of diameter, volume, density, thickness, surface area, Hounsfield unit standard deviation, or mean.

19. A system for multi-scheme analysis of patient-specific anatomical features from medical images, wherein the system comprises a server, The aforementioned server, Receiving a patient's medical image and metadata associated with the medical image showing a selected lesion, Using a segmentation algorithm, the medical image is automatically processed, the pixels of the medical image are labeled, and a score is generated indicating the likelihood that the pixels are correctly labeled. The method for classifying one or more patient-specific anatomical features in a medical image is to use an anatomical feature recognition algorithm to probabilistically match associated groups of labeled pixels against a set of anatomical knowledge data, wherein the set of anatomical knowledge data includes multiple groups of nodes representing various anatomical features, and the probabilistic matching of associated groups of labeled pixels against the set of anatomical knowledge data is performed by identifying the group of nodes in the set of anatomical knowledge data that is most similar to the associated groups of labeled pixels. To generate a 3D surface mesh model that defines the surface of one or more classified patient-specific anatomical features, Based on the selected lesion, information is extracted from the 3D surface mesh model. Based on the extracted information, generate physiological information associated with the selected lesion for the 3D surface mesh model, wherein the physiological information includes physiological measurements or characteristics related to the diagnosis and / or treatment of the selected lesion, calculated or determined from the extracted information. A system configured to perform the following actions.

20. The system according to claim 19, wherein the information extracted from the 3D surface mesh model comprises a 3D surface mesh model of anatomical features separated from one or more classified patient-specific anatomical features based on the selected lesion.

21. A non-transient computer-readable memory medium having instructions, wherein the instructions, when loaded by at least one processor, Receiving a patient's medical image and metadata associated with the medical image showing a selected lesion, Using a segmentation algorithm, the medical image is automatically processed, the pixels of the medical image are labeled, and a score is generated indicating the likelihood that the pixels are correctly labeled. The method for classifying one or more patient-specific anatomical features in a medical image is to use an anatomical feature recognition algorithm to probabilistically match associated groups of labeled pixels against a set of anatomical knowledge data, wherein the set of anatomical knowledge data includes multiple groups of nodes representing various anatomical features, and the probabilistic matching of associated groups of labeled pixels against the set of anatomical knowledge data is performed by identifying the group of nodes in the set of anatomical knowledge data that is most similar to the associated groups of labeled pixels. To generate a 3D surface mesh model that defines the surface of one or more classified patient-specific anatomical features, Based on the selected lesion, information is extracted from the 3D surface mesh model. Based on the extracted information, generate physiological information associated with the selected lesion for the 3D surface mesh model, wherein the physiological information includes physiological measurements or characteristics related to the diagnosis and / or treatment of the selected lesion, calculated or determined from the extracted information. A non-transient, computer-readable memory medium that causes at least one processor to perform the above.

Images