Method and system utilizing quantitative imaging

Abstract

Methods and systems utilizing quantitative imaging are presented herein. Advantageously, the systems and methods of the present disclosure utilize a hierarchical analysis framework that identifies and quantifies biological properties / analytes from imaging data, and then identifies and characterizes one or more pathologies based on the quantified biological properties / analytes. This hierarchical approach using imaging to examine underlying biology as an intermediate for assessing pathology provides numerous analysis and processing advantages over systems and methods configured to directly determine and characterize pathology from underlying imaging data.

Description

[0001] This application is a divisional application with application date of May 24, 2019, application number 201980049912.9, and invention name “Methods and systems using quantitative imaging”, all contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to quantitative imaging and analysis. More specifically, the present disclosure relates to systems and methods for analyzing pathology using quantitative imaging. Background Art

[0003] Imaging, especially when performed using safe and noninvasive methods, represents the most powerful method for locating the origin of disease, capturing its detailed pathology, guiding therapy, and monitoring health progression. Imaging is also an extremely valuable and low-cost method for mitigating these human and financial costs by allowing appropriate early intervention that is both inexpensive and disruptive.

[0004] Enhanced imaging techniques have made medical imaging an essential component of patient care. Imaging is particularly valuable because it provides spatially and temporally localized anatomical and functional information using non-invasive or minimally invasive methods. However, techniques are needed to effectively exploit increased temporal and spatial resolution to exploit patterns or features in data that are not easily assessed by the human eye and to manage large amounts of data in this way to effectively integrate them into clinical workflows. Without assistance, clinicians have neither the time nor the ability to effectively extract the available information content, and in any case typically interpret the information subjectively and qualitatively. Integration of quantitative imaging for individual patient management and clinical trials for therapy development requires new classes of decision support informatics tools to enable the medical community to fully exploit the capabilities that may be available with evolving and growing imaging modalities within the realities of existing workflows and reimbursement constraints.

[0005] Quantitative results from imaging methods have the potential to be used as biomarkers in routine clinical care and clinical trials, for example according to the widely accepted NIH Consensus Conference definition of biomarkers. In clinical practice, quantitative imaging aims to (a) detect and characterize disease before, during, or after a course of therapy, and (b) predict the course of disease with or without therapy. In clinical research, imaging biomarkers can be used to define endpoints for clinical trials.

[0006] Quantitative builds on the development of imaging physics, which has led to improvements in spatial, temporal and contrast resolution and the ability to excite tissues with multiple energies / sequences, resulting in different tissue-specific responses. These improvements can therefore allow for tissue differentiation and functional assessment, and are evident in, for example, computed tomography (CT), dual-energy computed tomography (DECT), spectral computed tomography (spectral CT), computed tomography angiography (CTA), cardiac computed tomography angiography (CCTA), magnetic resonance imaging (MRI), multi-contrast magnetic resonance imaging (multi-contrast MRI), ultrasound (US), and targeted or conventional contrast agent methods with various imaging modalities. Quantitative imaging measures specific biological properties that indicate the effectiveness of one treatment relative to another, how effective the current treatment is, or what risk a patient faces if he or she remains untreated. As a measuring device, a scanner combined with image processing of the formed image has the ability to measure the properties of tissues based on the physical principles associated with a given imaging method and how different tissues respond to it. While the image formation process varies widely across modalities, some generalizations help frame the overall assessment, and despite exceptions, nuances, and subtleties, the generalizations drive the true conclusions and are omitted until and unless they are seen as some of the biggest opportunities.

[0007] Imaging in the early stages of clinical testing of new therapeutic agents helps to understand the basic biological pathways and pharmacological actions. It can also reduce the cost and time required to develop new drugs and therapeutic agents. In the later stages of development, imaging biomarkers can serve as important endpoints of clinical benefit and / or serve as companion diagnostics for helping to prescribe and / or follow specific patient conditions for personalized therapy. In all stages, imaging biomarkers can be used to select patients based on disease state or stratify patients in order to better demonstrate the therapeutic effect.

[0008] a. Quantitative Medical Imaging:

[0009] Enhanced imaging techniques have made medical imaging an essential component of patient care. Imaging is particularly valuable because it provides spatially and temporally localized anatomical and functional information using non-invasive or minimally invasive methods. However, techniques are needed to handle the increased resolution to exploit patterns or features in the data that are not easily assessed by the human eye and to manage large amounts of data in such a way that they can be effectively integrated into the clinical workflow. With newer high-resolution imaging techniques, radiologists would be "drowned" in the data without assistance. The integration of quantitative imaging for individual patient management will require new classes of decision support informatics tools to enable the medical community to fully exploit the capabilities of these new tools within the realities of existing workflows and compensation constraints.

[0010] In addition, quantitative imaging methods are becoming increasingly important for (i) preclinical studies, (ii) clinical studies, (iii) clinical trials, and (iv) clinical practice. Imaging in the early stages of clinical testing of new therapeutic agents helps to understand the basic biological pathways and pharmacological effects. It can also reduce the cost and time required to develop new drugs and therapeutic agents. In the later stages of development, imaging biomarkers can serve as important endpoints for clinical benefit. In all stages, imaging biomarkers can be used to select patients based on disease state or stratify patients in order to better demonstrate the therapeutic effect.

[0011] Improving patient selection through the use of quantitative imaging will reduce the sample size required for a given trial (by increasing the fraction of evaluable patients and / or reducing the impact of superfluous variables) and help identify subpopulations that may benefit most from proposed treatments. This will reduce the time and cost of development of new drugs, but may also result in a corresponding reduction in the size of the "target" population.

[0012] Disease is not simple, and although disease often manifests itself locally, it is often systemic. Multifactorial assessment of objectively correlated tissue properties has proven to be an effective approach across medicine and is also true here, representing a panel or "spectrum" of continuous indicators, sometimes ideally proven to be future "surrogates" and / or difficult to measure but recognized endpoints. Computer-assisted measurement of lesion and / or organ biology and quantification of tissue composition in the first or second reader paradigm, which enables quantitative imaging assessment based on phenotypes implemented in an architecture that actively optimizes interoperability using modern clinical IT systems, is made possible by interdisciplinary convergence between next-generation computational methods for personalized diagnosis, providing clinicians with power as they manage their patients across a continuum of disease severity for improved classification across surgical, medical and surveillance pathways. More timely and accurate assessments at a level of granularity and complexity that is closer to the complexity of the disease itself rather than insisting on the assumption that it can be simplified to a level that hides the underlying biology will result in improved outcomes and more efficient use of healthcare resources, with benefits far outweighing the cost of the tool.

[0013] b. Phenotyping:

[0014] Radiological imaging is usually interpreted subjectively and qualitatively for medical conditions. The term phenotype is used in medical literature as the observable characteristic group of an individual caused by the interaction of its genotype with the environment. Phenotype usually means objectivity, that is, phenotype can be called real rather than subjective. Radiology is well known for its ability to visualize characteristics and it can be verified as an objective truth standard more and more (U.S. application serial number 14 / 959,732; U.S. application serial number 15 / 237,249; U.S. application serial number 15 / 874,474; and U.S. application serial number 16 / 102,042). Therefore, radiological images can be used to determine phenotypes, but usually lack sufficient methods for determining phenotypes.

[0015] Advantageously, phenotyping has a truth basis and can therefore be independently and objectively evaluated. Furthermore, phenotyping has become an accepted formalism in the healthcare industry for managing patient treatment decisions. Therefore, phenotypes have a high degree of clinical relevance. Finally, phenotyping is consumer relevant. This both allows for self-advocacy and acts as a motivating factor for lifestyle changes.

[0016] Early identification of phenotypes based on continuous metrics from comprehensive panels rather than as a test of a single feature alone will enable rapid intervention to prevent irreversible damage and death. Solutions are critical to fully preempting events or at least improving diagnostic accuracy at the onset of signs and / or symptoms. Patients at higher risk can be characterized using efficient workflow solutions that automatically measure structure and quantify tissue composition and / or hemodynamics, allowing patients at higher risk to be treated differently than those not at higher risk. There will be tremendous clinical implications if plaque morphological properties can be linked to embolic potential.

[0017] Imaging phenotypes can be correlated with gene expression patterns in association studies. This could have clinical impact, as imaging is routinely used in clinical practice, providing unprecedented opportunities to improve decision support in personalized therapy at low cost. Correlating specific imaging phenotypes with large-scale genomic, proteomic, transcriptomic and / or metabolomic analyses has the potential to influence treatment strategies by generating more definitive and patient-specific prognoses and measures of response to drugs or other therapies. However, the methods used to extract imaging phenotypes to date are mostly empirical in nature and are based primarily on human observations, albeit expert observations. These methods have embedded human variability and are clearly not scalable to support high-throughput analysis.

[0018] At the same time, the convergence of unmet needs to enable more personalized medicine without increasing costs - in fact, to better control costs and / or avoid unfortunate events by being proactive in preventive medicine, comparative effectiveness, reimbursement methods, etc., rather than reacting to unfortunate events, has created unprecedented pressure for technological advances to provide capabilities, and to deliver them in a way that simultaneously reduces costs.

[0019] In addition to the problem of phenotypic classification (classifying unordered categories), there is also the problem of outcome prediction / risk stratification (predicting ordered levels of risk). Both problems have clinical utility, but are the result of different technical device characteristics. In particular, one problem is not strictly dependent on the other.

[0020] Without limiting the generality, examples of phenotypic classifications with clinical relevance are provided below:

[0021] An example presentation of the "stable plaque" phenotype of atherosclerosis can be described as follows:

[0022] ● “Healing” disease with low response to intensive anti-statin regimens

[0023] Fewer balloon / stent complications

[0024] ●Sometimes stenosis > 50%

[0025] ●Sometimes Ca is higher and deeper

[0026] Minimal or no lipid, bleeding, and / or ulceration

[0027] ●Smooth appearance

[0028] Such plaques generally have lower adverse event rates than the "unstable plaque" phenotype.

[0029] ●Active disease that may respond well to intensive lipid-lowering and / or anti-inflammatory regimens

[0030] ●More complications of balloon / stent

[0031] ●Sometimes the stenosis is less than 50%

[0032] Low or diffuse calcium, calcium close to the lumen, napkin ring sign, and / or microcalcifications

[0033] ●Sometimes there is evidence of a higher lipid content and thin cap

[0034] Sometimes bleeding or signs of intraplaque hemorrhage (IPH) and / or ulceration

[0035] Such plaques have been reported to have adverse event rates 3 to 4 times higher than stable phenotypes. Both of these examples can be assessed in a single patient encounter, but other phenotypes such as "rapid progressors" can be determined by comparing the rate of change of a characteristic over time, i.e., a phenotype that exists statically at a certain point in time and is informed by kinetics and / or how things change n times.

[0036] c. Machine learning technology and deep learning technology:

[0037] Deep learning (DL) methods have been very successfully applied to many difficult machine learning (ML) and classification tasks originating from complex real-world problems. Notable recent applications include computer vision (e.g., optical character recognition, facial recognition, satellite image interpretation, etc.), speech recognition, natural language processing, medical image analysis (image segmentation, feature extraction and classification), discovery and validation of clinical and molecular data biomarkers. An attractive feature of the method is that it can be applied to both unsupervised and supervised learning tasks.

[0038] Neural Network (NN) and Deep NN methods, broadly including Convolutional Neural Network (CNN), Recursive Convolutional Neural Network (RCNN), etc., have been shown to be based on a good theoretical foundation and are generally modeled after the principle is regarded as representing the higher-level cognitive functions of the human brain. For example, in the neocortex, a brain region associated with many cognitive abilities, sensory signals propagate through a complex local modular hierarchy that learns to represent observations made over time - this is a design principle that has led to the general definition and construction of CNNs for image classification and feature extraction, namely. However, what is the more fundamental reason for the superior performance of DL networks and methods compared to frameworks with the same number of fitting parameters but without a deep hierarchical architecture is somewhat of an undecided question.

[0039] Conventional ML methods for image feature extraction and image-based learning from raw data have many limitations. Notably, spatial context is often difficult to obtain in ML methods using feature extraction when features are at a summary level rather than the 3D level of the raw image, or when the 3D level they are at is such that it is not linked to biological truth that can be objectively verified but is only a mathematical operator. The use of raw data sets that do contain spatial context often lacks objective baseline truth labels for the extracted features, but the use of raw image data itself contains a lot of variation that is "noise" relative to the classification problem at hand. In applications outside of imaging, this problem is often alleviated by very large training sets, so that the network training "learns" to ignore this variation and only focuses on significant information, but large training sets of this size are often not available in medical applications, especially due to cost, logistics, and privacy issues using baseline truth annotated data sets. The systems and methods disclosed herein help overcome these limitations.

[0040] d. Example applications: Cardiovascular measures such as fractional flow reserve (FFR) and plaque stability or high-risk plaques (HRP):

[0041] In the past 30 years, new treatments have been revolutionary in improving outcomes, but cardiovascular disease still imposes a burden of $320 billion per year on the U.S. economy. There are a large number of patient populations that can benefit from better characterization of the risk of major adverse coronary or cerebral events. The American Heart Association (AHA) has inferred an ACSD (atherosclerotic cardiovascular disease) risk score for a group project: 9.4% of the risk of adverse events in all adults (age>20) is greater than 20% in the next 10 years, and 26% of the risk is between 7.5% and 20%. Applying this to the group has obtained 23 million high-risk patients and 57 million moderate risk patients. The 80 million at risk can be compared with the 30 million American patients currently on statin therapy in the attempt to avoid new events or recurrences, and the 16.5 million diagnosed with CVD. Among the patients on statins, some patients will develop occlusive disease and acute coronary syndrome (ACS). The vast majority of patients are not aware of their disease progression before the onset of chest pain. Additional outcome and cost improvements in coronary artery disease will move from improving noninvasive diagnosis to identifying which patients have progressive disease on first-line therapy.

[0042] Cardiac health can be significantly affected by the degeneration of the arteries surrounding the heart. Multiple factors (e.g., tissue properties such as angiogenesis, neovascularization, inflammation, calcification, lipid deposition, necrosis, hemorrhage, stiffness, density, stenosis, dilation, remodeling rate, ulceration, blood flow (e.g., blood flow in a channel), pressure (e.g., pressure of blood in a channel or pressure of one tissue pressing against another), cell type (e.g., macrophages), cell arrangement (e.g., cell arrangement of smooth muscle cells), or shear stress (e.g., shear stress of blood in a channel), cap thickness, and / or tortuosity (e.g., entrance and exit angles)) can render these arteries less effective in delivering oxygen-filled blood to surrounding tissues. Fig.35 ).

[0043] Coronary artery function testing, mainly stress echocardiography and single photon emission computed tomography myocardial perfusion testing (SPECT MPI), is currently the main noninvasive method for diagnosing obstructive coronary artery disease. In the United States, more than 10 million functional tests are performed each year, with positive results driving 2.6 million visits to catheterization laboratories for invasive angiography to confirm the discovery of coronary artery disease.

[0044] Another way to assess perfusion is to determine the ability of the vasculature to transport oxygen. Specifically, the reduced ability can be quantified as the blood flow reserve fraction or FFR. FFR is not a direct measure of ischemia, but rather a substitute for measuring the ratio of pressure drop across the lesion. Changes in vessel diameter relative to other segments of the same vessel caused by local vasodilation damage at maximum hyperemia can produce significant hemodynamic effects, resulting in abnormal FFR measurements. During physical FFR measurements, the infusion of adenosine reduces downstream resistance to allow increased blood flow in hyperemic states. Physically measuring FFR requires an invasive surgical procedure involving a physical pressure sensor within the artery. Because this level of trauma can be risky and inconvenient in itself, a method for estimating FFR with high accuracy without the need for physical measurement is needed. The ability to perform this measurement non-invasively will also reduce significant "treatment bias": once the patient is in the catheterization laboratory, stent placement is relatively easy to perform, so many people have noticed that overtreatment occurs, but if the blood flow reserve can be assessed non-invasively, it may improve the decision on whether to implant a stent or not. Likewise, blood flow reserve also applies to perfusion of brain tissue (eg, related to hyperemia in the brain).

[0045] Known issues with functional testing are sensitivity and specificity. It is estimated that some 30-50% of patients with cardiovascular disease are misclassified and over- or under-treated, with significant monetary and quality of life costs. Functional testing is also expensive, time-consuming, and of limited use for patients with pre-obstructive disease. False positives from non-invasive functional testing are a significant factor in the overuse of both invasive coronary angiography and percutaneous coronary intervention in stable patients. Studies of the impact of false negatives estimate that of the 3.8 million annual MPI tests given to U.S. patients suspected of having coronary dynamic disease (CAD), nearly 600,000 will report false negative results resulting in 13,700 acute coronary events, many of which will be preventable by the introduction of appropriate medical therapy. Another shortcoming of functional testing is that it is transient in nature: ischemia is a lagging indicator that follows the anatomical changes brought about by disease progression. Patients at high risk for ACS can be better served if intensive medical therapy can be used to detect and reduce future culprit lesions before the onset of ischemia.

[0046] Coronary computed tomography angiography (CCTA), especially when used in tandem with quantitative analysis software, is evolving as the ideal test modality to fill the gap in understanding the extent and rate of progressive coronary artery disease. Over the past 10 years, the CT scanner fleet in most countries has been upgraded to higher speed, higher detector count machines capable of excellent spatial resolution without slowing the heart or extensive breath holding. Radiation doses have been greatly reduced to a point equal to or lower than that of SPECT MPI and invasive angiography.

[0047] Recent analyses of data from landmark trials such as SCOT-HEART, PREDICT and PROMISE, as well as others, have demonstrated the value of using CCTA to detect non-obstructive disease, sometimes referred to as high-risk plaque (HRP) or vulnerable plaque, by identifying patients at increased risk for future adverse events. Study designs vary and include nested case-control cohorts comparing CCTA-enrolled patients who experience cardiovascular (CV) events to controls with similar risk factors / demographics, comparisons to FFR, and multi-year follow-up of the large "Test and Treat" study. NICE's recent favorable determination to position CCTA as a frontline diagnostic was based on a significant reduction in CV events in the CCTA arm of the SCOT-HEART study, which was attributed to changes in plaque findings at the time of initiation of medical therapy or by CCTA.

[0048] Based on the PROMISE findings indicating the greatest need for plaque assessment, important target patient groups are those with stable chest pain and no prior history of CAD with typical or atypical angina symptoms (based on SCOT-HEART data), as well as those with non-obstructive disease (<70% stenosis) and younger patients (e.g., 50-65 year old group). Patients with non-obstructive CAD who have a high-risk plaque profile based on CCTA analysis can be assigned to the most appropriate high-intensity statin therapy (especially when considering decisions about very expensive new lipid-lowering therapies such as PCSK9 inhibitors or anti-inflammatory drugs such as canakinumab), or longitudinal follow-up with the addition of new antiplatelet agents to mitigate the risk of coronary thrombosis and / or possible intensification or de-escalation of therapy. CCTA is an ideal diagnostic tool because it is non-invasive and requires less radiation than cardiac catheterization.

[0049] The pathological literature on culprit lesions implicated in fatal heart attacks indicates that clinically non-obstructive CAD is more likely to be home to the majority of high-risk plaques than more occlusive plaques, which tend to be more stable. Recent studies have confirmed these findings by identifying culprit lesions from ACS patients undergoing invasive angiography and comparing them to precursor plaques in baseline CCTA. In a cohort undergoing clinically indicated CCTA, it was found that patients with non-obstructive CAD, 38% of those so tested, remained at significant risk for medium- to long-term major adverse cardiovascular and / or cerebrovascular events (MACCE). The hazard ratio based on the number of diseased segments, independent of obstruction, was found to be a significant long-term predictor of MACCE in this cohort. A contributing factor to the predictive value of clinically non-obstructive CAD is that these lesions are more likely to be home to the majority of high-risk plaques than more occlusive plaques, which tend to be more stable.

[0050] Further confirmation of the potential utility of CCTA in the detection and management of obstructive and pre-occlusive atherosclerotic lesions has been seen in several recently published longitudinal studies of the effects of statin and anti-inflammatory drug therapy, where plaque remodeling to a more stable representation and plaque regression were observed in the treated branches. This confirms the body of earlier studies of intravascular ultrasound (IVUS, sometimes utilizing "virtual histology" VH), near infrared spectroscopy (NIRS), optical coherence tomography (OCT), etc., that have explored disease progression and treatment effects under a variety of lipid-lowering drug regimens. Recent drug trials provide potential plaque biomarkers for demonstrating the efficacy of new medical therapies. The Integrated Biomarker and Imaging Study-4 (IBIS-4) found that calcification progressed with the potential protective effects of statins. Other studies found that lipid-rich necrotic cores (LRNC) were reduced under statin therapy. In these studies, clinical variables, when used alone, poorly distinguished the characteristics of high-risk plaques. These studies emphasize the importance of complete characterization and assessment of the entire coronary tree rather than just the culprit lesion to allow for more accurate risk stratification that can be appropriately analyzed with CCTA. In a meta-analysis, CCTA had good diagnostic accuracy for detecting coronary plaques compared to IVUS with small differences in assessing plaque area and volume, percentage of stenosis area, and a slight overestimation of lumen area. It was also found that increasing the rate of change of lipid-rich necrotic cores and their distance from the lumen had high prognostic value. In addition, the results of the ROMICAT II trial showed that for stable CAD patients with acute chest pain but negative initial ECG and troponin, identifying high-risk plaques on CCTA increased the likelihood of ACS, which was independent of significant CAD and clinical risk assessment. Examinations performed by CCTA have been established for the evaluation of coronary atherosclerotic plaques. For patients in whom the necessity of invasive procedures is uncertain, it would be beneficial and feasible to non-invasively predict MACCE using CCTA, which gives an overall estimate of the disease burden and risk of future events.

[0051] The prevalence of carotid artery disease and CAD are closely related. Carotid atherosclerosis has been shown to be an independent predictor of MACCE, even in patients without pre-existing CAD. Such findings indicate a common underlying pathogenesis shared in both conditions, which is further supported by the Multi-Ethnic Study of Atherosclerosis (MESA). Atherosclerosis progresses through the progression of arterial wall lesions, leading to the accumulation and inflammatory response of cholesterol-rich lipids. These change similarly (even if not identical) in coronary arteries, carotid arteries, aortas, and peripheral arteries. Certain plaque characteristics such as large artery atherosclerotic cores and lipid-rich content, thin caps, outward remodeling, plaques infiltrated by macrophages and lymphocytes, and culture medium thinning easily induce fragility and rupture.

[0052] e. Non-invasive determination of HRP and / or FFR:

[0053] Noninvasive assessment of the functional significance of stenosis using CCTA is of clinical and economic interest. The combination of lesion or vessel geometry or anatomy with the characteristics or composition of the tissues comprising the wall and / or plaque in the wall, collectively referred to as plaque morphology, may explain the outcome of lesions with higher or lower risk plaques (HRP) and orthogonal considerations of normal and abnormal flow reserve (FFR). Lesions with large necrotic cores may develop dynamic stenoses due to outward remodeling during plaque formation, resulting in more tissue extension, tissue stiffening, or the smooth muscle layer has been extended to the limit of the Glagov phenomenon, after which the lesion gradually invades the lumen itself. Similarly, inflammatory damage and / or oxidative stress may lead to local endothelial dysfunction manifested as impaired vasodilation.

[0054] If the tissue that makes up the plaque is primarily stroma or "fatty streaks" that are not organized into a necrotic core, the plaque will expand adequately to keep up with demand. However, if the plaque has a larger necrotic core, it will not be able to expand. The blood supply will not be able to keep up with demand. Plaque morphology will improve accuracy by evaluating complicating factors such as LRNC, calcification, hemorrhage, and ulceration with an objective truth that can be used to validate the underlying information in a way that other methods cannot do due to the lack of intermediate measurement target validation.

[0055] But that's not all plaques can do. Often, plaques will actually rupture, suddenly causing a clot that can then cause infarction of heart or brain tissue. Plaque morphology can also identify and quantify these high-risk plaques. For example, plaque morphology can be used to determine how close the necrotic core is to the lumen: a key determinant of infarction risk. Knowing whether a lesion restricts blood flow under stress does not indicate the risk of rupture, and vice versa. Other methods, such as computational fluid dynamics (CFD), can simulate blood flow restriction but not infarction risk without objectively validated plaque morphology. The fundamental advantage of plaque morphology is that its accuracy depends on the determination of vascular structure and tissue properties, while allowing the phenotype to be determined.

[0056] There is an increasing amount of clinical guidance on the optimal management of patients using different assessments of blood flow reserve. It is well known that obstructive lesions with high-risk features (large necrotic core and thin cap) portend the greatest likelihood of future events and, importantly, the reverse is also true.

[0057] Methods for determining FFR using CFD have been published without accurately assessing plaque morphology. However, CFD-based flow reserves consider only the lumen, or at best, the changes in the lumen surface at different parts of the cardiac cycle. Considering only the lumen surface at best requires processing both systole and diastole to obtain motion vectors (which even most available methods do not do), but even without considering what happens under stress, this is true because these analyses are performed using computer simulations of what might happen under stress rather than measuring actual stress, and are not based on actual properties that cause vasodilation but on blood passage alone. Some methods attempt to simulate forces and apply biomechanical models, but the methods use assumptions rather than validated measurements of wall tissue. Therefore, these methods cannot anticipate what might happen if stress actually causes rupture beyond the basic assumptions. In contrast, characterizing the tissue can address these issues. Wall properties, including the effect on the vasodilation capacity of the vessel due to its wall's distensibility, are considered superior because the composition of the lesion determines flexibility and energy absorption, stable lesions are still overtreated and the assessment of MACCE risk is incomplete. Advantages of using morphology to assess FFR include the fact that morphology is a leading indicator, FFR lags, and the presence and extent of HRP will better inform treatment of marginal subjects. The importance of accurate assessment by morphology resolution is reinforced by studies increasingly showing that morphology can predict FFR but FFR does not predict morphology. That is, a validly assessed morphology can not only determine FFR, but also the discrete changes in plaque that may shift a patient from ischemia to infarction or HRP. Summary of the invention

[0058] Provided herein are systems and methods that utilize a hierarchical analysis framework to identify and quantify biological properties / analytes from imaging data, and then identify and characterize one or more medical conditions based on the quantified biological properties / analytes. In some embodiments, the systems and methods merge computer image analysis and data fusion algorithms with patient clinical chemistry and blood biomarker data to provide a multifactorial panel that can be used to distinguish between different subtypes of the disease. Therefore, the systems and methods of the present disclosure can advantageously implement biological and clinical insights in advanced computational models. These models can then be interfaced with complex image processing by specifying rich ontologies associated with an increasing understanding of pathogenesis, and take the form of rigorous definitions of what is measured, how it is measured and evaluated, and how it is clinically relevant to subtypes and stages of the disease that can be verified.

[0059] Human diseases exhibit robust phenotypic differences that can be understood by applying complex classifiers to extracted features that capture spatial, temporal, and spectral outcomes that are measurable by imaging but difficult to understand without assistance. Traditional computer-aided diagnosis makes inferences based on image features in only a single step. In contrast, the systems and methods of the present disclosure employ a hierarchical inference scheme that includes intermediate steps of determining spatially resolved image features and temporally resolved dynamics at multiple levels of biological target components of morphology, composition, and structure, which are then used to derive clinical inferences. Advantageously, the hierarchical inference scheme ensures that the clinical inferences can be understood, validated, and interpreted at each level of the hierarchy.

[0060] Thus, in an example embodiment, the systems and methods of the present disclosure utilize a hierarchical analysis framework comprising a first level of algorithms measuring biological properties that can be objectively verified against a truth standard independent of imaging, and a subsequent second set of algorithms for determining a medical or clinical condition based on the measured biological properties. For example, this framework is applicable to a variety of different biological properties, such as angiogenesis, neovascularization, inflammation, calcification, lipid deposition, necrosis, hemorrhage, rigidity, density, stenosis, dilation, remodeling rate, ulceration, blood flow (e.g., blood flow of blood in a channel), pressure (e.g., pressure of blood in a channel or pressure of one tissue on another tissue), cell type (e.g., macrophages), cell arrangement (e.g., cell arrangement of smooth muscle cells) or shear stress (e.g., shear stress of blood in a channel), cap thickness and / or tortuosity (e.g., entrance angle and exit angle) in an "and / or" manner, i.e., individually or in combination. The measured variables of each of these properties can be measured, such as the quantity and / or degree and / or trait of the property. Example conditions include perfusion / ischemia (e.g., limited) (e.g., perfusion / ischemia of brain or cardiac tissue), perfusion / infarction (e.g., complete resection) (e.g., perfusion / infarction of brain or cardiac tissue), oxygenation, metabolism, blood flow reserve (perfusion capacity), malignancy, invasion, and / or risk stratification (whether as probability of event or as time to event (TTE)), such as major adverse cardiovascular or cerebrovascular events (MACCE). The truth base can include, for example, biopsy, expert tissue annotation forming the resected tissue (e.g., endarterectomy or autopsy), expert phenotypic annotation of the resected tissue (e.g., endarterectomy or autopsy), physical pressure lines, other imaging modalities, physiological monitoring (e.g., ECG, SaO2, etc.), genomics and / or proteomics and / or metabolomics and / or transcriptomics assays, and / or clinical outcomes. These properties and / or conditions can be measured at a given time point and / or can vary across time (longitudinal).

[0061] In an exemplary embodiment, the systems and methods of the present application advantageously relate to computer-assisted phenotyping (CAP) of diseases. CAP is a new and exciting addition to the field of computer-aided diagnosis (CAD). As disclosed herein, CAP can apply hierarchical inference that incorporates computer image analysis and data fusion algorithms to patient clinical chemistry and blood biomarker data to provide a measured multi-factor panel or "spectrum" that can be used to distinguish between different subtypes of diseases that will be treated differently. Therefore, CAP implements new methods for robust feature extraction, data integration, and scalable computing strategies to implement clinical decision support. For example, the space-time texture (SpTeT) method captures relevant statistical features for characterizing tissues spatially and dynamically. Spatial features are mapped to, for example, characteristic patterns of lipids mixed with extracellular matrix fibers, necrotic tissue, and / or inflammatory cells. Dynamic features are mapped to, for example, endothelial magnetic permeability, neovascularization, necrosis, and / or collagen decomposition.

[0062] Compared to current CAD methods that perform clinical inference in a single step of machine classification based on image features, the system and method of the present application can advantageously utilize a hierarchical inference scheme that can be applied not only to spatially resolved image features at the beginning, but also to temporally resolved dynamics at multiple layers of biological target components of morphological and structural composition in the middle, and then to the final clinical inference. This produces a system that can be understood, verified, and interpreted at each layer in the hierarchy, from low-level image features at the bottom to biological and clinical features at the top.

[0063] The disclosed system and method have been improved phenotypic classification and outcome prediction. Phenotypic classification can occur on two layers, on the one hand, a single anatomical position, and on the other hand, a more generally described body part. The input data of the former can be a 2D data set and the input data of the latter can be a 3D data set. However, for phenotypic classification, objective truth can be on any layer, and for outcome prediction / risk stratification, objective truth usually occurs on the patient layer, but in some cases objective truth may be more specific (for example, on which side of the body the symptoms of stroke are manifested). The meaning here is that the same input data can be used for two purposes, but the model will be significantly different due to the layer used by the input data and the basis of the truth annotation.

[0064] Although it is possible to perform model building read vectors as input data, performance is typically limited by the implemented measured variables. The present application advantageously utilizes unique measured variables (e.g., cap thickness, calcium depth, and ulcers) to improve performance. Therefore, only the read vector method can be applied when the vector contains these measured variables (e.g., in combination with conventional measured variables). However, the systems and methods of the present disclosure can advantageously utilize deep learning (DL) methods, which can provide even richer data sets. The systems and methods of the present application can also advantageously utilize unsupervised learning applications, thereby providing better scalability across data domains (a very highly desired feature considering the speed at which new biological data is generated).

[0065] In the example embodiments presented herein, convolutional neural networks (CNNs) can be used to establish classifiers in a method that can be characterized as transfer learning using a fine-tuning method. A CNN trained on roughly minimal imaging data on a powerful computing platform can be successfully used to classify images that have not yet been annotated in network training. This is intuitively understandable because many common feature classifications help identify images of very different objects (i.e., shapes, boundaries, orientations in space, etc.). It is then conceivable that a CNN trained to identify thousands of different objects using a pre-annotated data set of tens of millions of images will perform basic image recognition tasks better than by chance, and will have performance comparable to a CNN trained from the beginning after a relatively small, slightly adjusted last classification layer sometimes referred to as a 'softmax layer. Because these models are very large and have been trained on a large number of pre-annotated images, they tend to "learn" very unique, distinctive imaging features. Therefore, a convolutional layer can be used as a feature extractor, or a trained convolutional layer can be slightly adjusted to adapt to the problem at hand. The first method is called transfer learning, and the latter is called fine-tuning.

[0066] CNNs are good at performing many different computer vision tasks. However, CNNs have some disadvantages. Two important disadvantages of importance to medical systems are 1) the need for large training and validation data sets, and 2) intermediate CNN calculations do not represent any measurable properties (sometimes criticized as "black boxes" whose principles are not described). The method disclosed herein can advantageously utilize a pipeline consisting of one or more stages that are individually biomeasurable and independently verifiable, followed by a convolutional neural network that starts with these stages rather than just the original image. In addition, certain transformations can be applied to reduce changes that are irrelevant to the problem at hand, such as unfolding a cross-section of a donut-shaped blood vessel before feeding the network to become a rectangular representation with a normalized coordinate system. These front-end pipeline stages simultaneously alleviate both of the two disadvantages of using CNNs for medical imaging.

[0067] Typically, the early convolutional layers act as feature extractors of increasing specificity, and the fully connected one or two final layers act as classifiers (e.g., "softmax layers"). Schematic representations of the sequence of layers in a typical CNN and their functions are available from many sources.

[0068] Advantageously, the systems and methods of the present disclosure utilize enriched data sets to enable non-invasive phenotyping of tissues determined by radiological data sets. One type of enrichment is to pre-process the data to perform tissue classification and use a "pseudo-color" overlay to provide a data set that can be objectively verified (as opposed to using only the original image, which does not have this possibility). Another type of enrichment is to use a transformation on the coordinate system to emphasize biologically plausible spatial context while removing noise variations to improve classification accuracy, allow for smaller training sets, or both.

[0069] In an example embodiment, the systems and methods of the present application may employ a multi-stage pipeline: (i) semantic segmentation for identifying and classifying regions of interest (e.g., which may represent quantitative biological analytes), (ii) spatial unfolding for converting cross-sections of tubular structures (e.g., venous / arterial cross-sections) into rectangles; and (iii) application of a trained CNN for reading the annotated rectangles and identifying which phenotype they belong to (e.g., stable or unstable plaque and / or normal or abnormal peak FFR) and / or predicted time of event (TTE). It should be noted that by training and testing the CNN on an unfolded dataset (in the case of unfolding) versus a donut-shaped dataset (in the case of unfolding), it can be demonstrated that unfolding improves the validation accuracy of each specific embodiment. Therefore, various embodiments of imaging tubular structures (e.g., plaque phenotyping) or other structures (e.g., lung cancer mass subtyping) or other applications may similarly benefit from performing similar steps (e.g., semantic segmentation followed by a spatial transformation such as unfolding (before applying the CNN)). However, it is contemplated that in some alternative embodiments, an untransformed dataset (eg, a dataset that is not spatially spread out) may be used to determine a phenotype (eg, in conjunction with or independent of the untransformed dataset).

[0070] In an example embodiment, semantic segmentation and spatial transformation may involve the following: the image volume may be preprocessed, the preprocessing including target initialization, normalization, and any other desired pre-compression, such as deblurring or restoration to form a region of interest containing the physiological target to be phenotyped. It is noteworthy that the region of interest may be the volume consisting of cross-sections through the volume, which is a body part that can be automatically determined or explicitly provided by the user. Targets of body parts that are tubular in nature may be accompanied by a centerline. The centerline may have branches when present. Branches may be marked automatically or by the user. Note that a generalization of the concept of the centerline may represent anatomy that is not tubular but benefits from some structural orientation, such as the region of a tumor. In any case, a centroid may be determined for each cross-section in the volume. For tubular structures, the centroid may be a channel, for example, the center of the lumen of a blood vessel. For lesions, the centroid may be the center of the mass of a tumor. The (optionally deblurred or restored) image can be represented in a Cartesian data set, in which x is used to represent how far away from the centroid, y represents the rotation θ, and z represents the cross section. Each branch or region will form one such Cartesian set. When multiple sets are used, "null" values ​​can be used for overlapping regions, that is, each physical voxel can be represented only once in the entire set in a way that fits together geometrically. Each data set can be paired with another data set having a sub-region labeled by an objectively verifiable tissue composition. Example labels for vascular tissue can be lumen, calcification, LRNC, IPH, etc. Example labels for lesions can be necrotic, neovascularization, etc. These labels can be objectively verified, for example, by histology. The paired data set can be used as input for the training step of establishing a convolutional neural network. In an example embodiment, two layers of analysis can be supported, one layer in a separate cross-sectional layer and the second layer in a volume layer. The output label represents a phenotype or risk stratification.

[0071] Exemplary image preprocessing may include deblurring or restoration using, for example, a patient-specific point spread determination algorithm to mitigate artifacts or image limitations caused by the image formation process. These artifacts and image limitations may reduce the ability to determine characteristics that predict phenotypes. For example, deblurring or restoration may be achieved by, for example, iteratively fitting a physical model of the scanner point spread function using regularized assumptions about the true latent density of different regions of the image.

[0072] In example embodiments, the CNN may be an AlexNet, Inception, CaffeNet, or other network. In some embodiments, the CNN may be reconfigured, for example, using the same number and type of layers, but with input and output dimensions changed (e.g., changing aspect ratio). Example implementations of various example CNNs are provided as open source, for example, on TensorFlow and / or in other frameworks, and may be used as open source and / or licensed configurations.

[0073] In example embodiments, the data set may be augmented. For example, in some embodiments, a 2D or 3D rotation may be applied to the data set. Thus, in the case of an untransformed (e.g., donut-shaped) data set, augmentation may involve, for example, a combination of random longitudinal flipping of the data set and random rotation of the data set (e.g., rotation of a random angle between 0 and 360 degrees). Similarly, in the case of a transformed (e.g., unfolded) data set, augmentation may involve, for example, a combination of random longitudinal flipping and random "rolling" of the image, such as a random number of pixels ranging from 0 to the width of the image (where rolling is similar to rotating around θ).

[0074] In an example embodiment, the data set can be enriched by using different colors to represent different tissue analyte types. These colors can be selected to visually contrast with respect to each other and with respect to non-analyte surfaces (e.g., normal walls). In some embodiments, non-analyte surfaces can be depicted in gray. In an example embodiment, data set enrichment can cause a reference truth annotation to tissue characteristics (e.g., tissue characteristics indicating plaque phenotypes), and a spatial context (e.g., taking an axis orthogonal to the blood vessel) of how such tissue characteristics are presented in a cross section can be provided. Such spatial context can include a coordinate system (e.g., based on polar coordinates relative to the centroid of the cross section), which provides a common basis for data set analysis relative to histological cross sections. Therefore, the enriched data set can be advantageously superimposed on the top of the color-coded pathologist annotation (and vice versa). Advantageously, the histology based on the annotated data set can then be used for training (e.g., CNN training) in combination with or independent of the image feature analysis of the radiological data set. It is noteworthy that histology based on annotated datasets can improve the efficiency of DL methods because histology based on annotated datasets uses relatively simple pseudo-color images instead of high-resolution full images without losing spatial context. In an example embodiment, the coordinate direction can be represented internally using unit phasors and phasor angles. In some embodiments, the coordinate system can be normalized, for example, by normalizing the radial coordinates relative to the wall thickness (such as to provide a common basis for comparing tubular structures / cross-sections of different diameters / thicknesses). For example, the normalized radial distance has a value of 0 at the inner boundary (inner wall lumen boundary) and a value of 1 at the outer boundary (outer wall boundary). It is noteworthy that this can be applied to tubular structures related to blood vessels or other pathophysiology (e.g., the gastrointestinal tract).

[0075] Advantageously, the enriched data set of the present application provides a non-invasive image-based classification (e.g., where a tissue classification scheme can be used to non-invasively determine a phenotype), which is based on a known reference truth. In some embodiments, the known reference truth can be non-radiological (e.g., histology or another ex vivo-based tissue analysis). Thus, for example, a radiological data set annotated to contain ex vivo reference truth data (e.g., histology information) can be advantageously used as input data for a classifier. In some embodiments, when annotating an enriched data set, multiple different known reference truths can be used in conjunction with each other or independently of each other.

[0076] As described herein, in some embodiments, the enriched data set can utilize a normalized coordinate system to avoid non-correlated changes associated with, for example, wall thickness and radial representation. In addition, as described herein, in an example embodiment, for example, before classification training (e.g., using CNN) and / or before running a training classifier on a data set, a "donut-shaped" data set can be "unfolded". It is noteworthy that in such embodiments, the analyte annotation of the training data set can be before transformation, such as after expansion or a combination of the two. For example, in some embodiments, an untransformed data set can be annotated (e.g., using ex vivo classification data such as histological information), and then transformed for classifier training. In such embodiments, the finer granularity based on ex vivo classification can collapse to match the lower expected granularity for in vivo radiological analysis, to reduce computational complexity, while addressing criticisms of "black box" what will be otherwise open.

[0077] In some embodiments, the colors and / or axes used to visualize the annotated radiological data set can be selected to correspond to the same colors / axes as typically presented in classification based on ex vivo reference truth (e.g., the same colors / axes used in histology). In an example embodiment, a transformed enriched data set (e.g., normalized for wall thickness) can be presented, wherein each analyte is visually represented with different contrasting colors and relative to background areas (e.g., black or gray) for all non-analyte areas. It is worth noting that, depending on the embodiment, the common background may or may not be annotated, and therefore may or may not be visually distinguished between non-analyte areas inside and outside the vessel wall or between background features (e.g., luminal surface irregularities, varying wall thickness, etc.). Therefore, in some embodiments, the annotated analyte areas can be visually depicted (e.g., color-coded and normalized for wall thickness) relative to a uniform (e.g., completely black, completely gray, completely white, etc.) background. In other embodiments, annotated analyte regions (e.g., color-coded for wall thickness but not normalized) can be visually depicted relative to annotated background (e.g., where different shades (gray, black, and / or white) can be used to distinguish between: (i) a central lumen region inside the inner lumen of a tubular structure, (ii) a non-analyte region inside the wall, and / or (iii) a region outside the outer wall). This can enable analysis of changes in wall thickness (e.g., due to ulcers or thrombi). In further example embodiments, the annotated dataset can include identification (and visualization) of regions of, for example, intraplaque hemorrhage and / or other morphological aspects. For example, regions of intraplaque hemorrhage can be viewed in red, LRNC in yellow, etc.

[0078] A specific embodiment of the systems and methods of the present application may be in guided vascular therapy. Classification may be established based on the likely dynamic behavior of plaque lesions (based on their physical properties or specific mechanisms, such as inflammation or cholesterol metabolism) and / or based on the progression of the disease (e.g., early versus late stages of its natural history). Such classifications may be used to guide patient treatment. In an example embodiment, the Stary plaque typing system employed by the AHA may be used as the basis for the types determined in vivo as shown in color overlays. An example mapping is ['I', 'II', 'III', 'IV', 'V', 'VI', 'VII', 'VIII'], resulting in class_map = [subclinical, subclinical, subclinical, subclinical, unstable, unstable, stable, stable]. However, the systems and methods of the present disclosure are not associated with Stary. As another example, the Virmani system ["calcified nodule", "CTO", "FA", "FCP", "healing plaque rupture", "PIT", "IPH", "rupture", "TCFA", "ULC"] has been used with class-map = [stable, stable, stable, stable, stable, stable, unstable, unstable, unstable, unstable], and other classification systems may yield similar high performance. In example embodiments, the systems and methods of the present disclosure may incorporate entirely different classification systems, may change class maps, or other variations. For FFR phenotypes, values ​​such as normal or abnormal may be used and / or quantities may be used to facilitate comparison with, for example, physical FFR.

[0079] Thus, in an example embodiment, the systems and methods of the present disclosure can provide a phenotypic classification of plaques based on an enriched radiological data set. In particular, one or more phenotypic classifications can include distinguishing stable plaques from unstable plaques, for example, where the ground truth basis for the classification is based on factors such as (i) lumen narrowing (which may be enhanced by additional measures such as tortuosity and / or ulceration), (ii) calcium content (which may be enhanced by depth, shape, and / or other complex representations), (iii) lipid content (which may be enhanced by cap thickness and / or other complex representations), (iv) anatomical structure or geometry, and / or (v) IPH or other content. Notably, this classification has been shown to have high overall accuracy, sensitivity, and specificity as well as a high degree of clinical relevance (potentially changing the existing standard of care for patients undergoing catheter and cardiovascular care).

[0080] Another example embodiment is lung cancer, where subtypes of masses can be determined to guide the most likely beneficial treatment for the patient based on the distinct phenotype. In particular, preprocessing and dataset enrichment can be used to separate solid versus semi-solid ("ground glass") sub-regions that differ in their degree of malignancy and indicate different optimal treatments.

[0081] In additional example embodiments, the systems and methods of the present disclosure may provide image preprocessing, image denoising, and novel geometric representations (e.g., affine transformations) of CT angiography (CTA) diagnostic images to facilitate and maximize the performance of deep learning algorithms based on CNNs for developing state-of-the-art classifiers and markers representing risk of adverse cardiovascular effects during the procedure. Thus, as disclosed herein, image deblurring or restoration may be used to identify lesions of interest and quantitatively extract plaque composition. Additionally, a transformation of images segmented in cross-section (along the main axis vessel) into, for example, an "unfolded" rectangular reference frame that follows the established lumen along the X-axis may be applied to provide a normalized frame to allow the DL method to best learn representative features.

[0082] Although the example embodiments herein utilize 2D annotated cross-sections for analysis and phenotyping, it should be noted that the present application is not limited to such embodiments. In particular, some embodiments may utilize, for example, enriched 3D data sets instead of or in addition to processing 2D cross-sections alone. Thus, in example embodiments, video interpretation based on computer vision may be applied to the classifier input data set. Note that processing multiple cross-sections sequentially, as along a centerline in a "movie" sequence, can generalize these methods for tubular structures, for example, moving the centerline and / or other 3D representations up and down according to aspects that best suit the anatomy.

[0083] In another example embodiment, the pseudo color representation in the enriched data set can have continuous values ​​across pixel or voxel positions. This can be used for "radiomics" features (with or without explicit validation) or verified tissue types calculated independently for each voxel. This set of values ​​can exist in any number of preprocessed overlays and can be fed into a phenotype classifier. It is worth noting that in some embodiments, the value of each pixel / voxel can be for any number of different features (e.g., for different analytes, it can be represented by any number of different overlays, which is sometimes referred to as "multiple occupancy"). Alternatively, each pixel / voxel can be assigned to only one analyte (e.g., assigned to only a single analyte overlay). In addition, in some embodiments, the pixel / voxel value of a given analyte (e.g., a given analyte) can be based on an all-or-nothing classification scheme (e.g., a pixel is calcium or not calcium). Alternatively, the pixel / voxel value of a given analyte can be a relative value, e.g., a probability score. In some embodiments, the relative values ​​of pixels / voxels are normalized across a set of analytes (e.g., so that the total probability adds up to 100%).

[0084] In an example embodiment, a classification model may be trained by applying, in whole or in part, multiscale modeling techniques such as partial differential equations to, for example, represent possible cell signaling pathways or plausible biological motivations.

[0085] Other alternative embodiments include using change data collected, for example, from multiple time points, rather than (only) data from a single time point. For example, if the amount or property of a negative cell type increases, this can be referred to as a "progressor" phenotype, while a "regressor" phenotype is for a decrease. A regressor may be, for example, due to a response to a drug. Alternatively, if the rate of change of, for example, LRNC is fast, this may suggest a different phenotype, such as a "rapidly progressor".

[0086] In some embodiments, non-spatial information, such as information derived from other assays (e.g., laboratory results) or demographic / risk factors or other measurements extracted from radiological images, can be fed to the final layer of the CNN to combine the spatial information with the non-spatial information.

[0087] It is noteworthy that although the systems and methods focus on phenotype classification, similar methods can be applied to outcome prediction. Such classification can be based on the baseline truth historical results assigned to the training data set. For example, the systems and methods of the present application can be used to determine life expectancy, quality of life, treatment efficacy (including comparison of different treatment methods), and other outcome predictions.

[0088] Examples of the systems and methods of the present application are further illustrated in the following multiple figures and detailed descriptions.

[0089] In further example embodiments, the systems and methods of the present disclosure provide for determination of fractional blood flow reserve in myocardial and / or cerebral tissues through measurement of plaque morphology. The systems and methods of the present disclosure may use sophisticated methods to characterize the vasodilatory capacity of a vessel through objectively validated determination of tissue types and properties that influence its distensibility. In particular, plaque morphology may be used as input to analysis of the dynamic behavior of the vascular system from a blood flow reserve perspective (training the model with blood flow reserve ground truth data). It is therefore possible to determine the dynamic behavior of the system rather than (only) a static description. Stenosis itself is notoriously of low predictive power as it only provides a static description, and accurate plaque morphology needs to be added for the highest accuracy imaging-based assessment of dynamic function. The present disclosure provides systems and methods for determining accurate plaque morphology and then processing to determine dynamic function.

[0090] In an example embodiment, deep learning is utilized to preserve the spatial context of tissue properties and vascular anatomy (collectively referred to as plaque morphology) at an optimal level of granularity, thereby avoiding excessive non-material variability in the training set while maintaining other simpler uses that require going beyond machine learning. Other alternative methods use measurements of only vascular structure, rather than a more complete treatment of tissue properties. Such methods can capture lesion length, stenosis, and possible inlet and outlet angles, but they ignore the determinants of vasodilation capacity. High-level assumptions about the flexibility of the arterial tree as a whole must be made to use these models, but plaque and other tissue properties result in the expansibility of the coronary tree being expansible in a heterogeneous manner. Different parts of the tree are more or less expansible. Since expansibility is a key factor in determining FFR, these methods are inadequate. Other methods that attempt to perform tissue properties do so without objective validation of their accuracy and / or without maintaining the data enrichment methods required for deep learning of medical images (e.g., transformations such as unfolding and validated pseudo-color tissue type overlays) that are optimal to provide the spatial context necessary for the effectiveness of deep learning methods. Some approaches attempt to increase the training set size by using synthetic data, but this is ultimately limited by the limited data upon which the synthetic generation is based, and is more limited by the data augmentation scheme than the amount of actual expansion of the input training set. Additionally, the systems and methods of the present disclosure are able to produce continuous assessments across the length of the vessel.

[0091] The systems and methods of the present disclosure effectively utilize objective tissue characterization validated by histology across multiple arterial beds. With regard to the relevance of the example application to atherosclerosis, plaque composition is similar in the coronary and carotid arteries, regardless of their age, and this will largely determine the relative stability, indicating that the representation at CCTA is similar to that at CTA. Minor differences in the range of various plaque characteristics can include thicker caps and a higher prevalence of intraplaque hemorrhage and calcified nodules in the carotid arteries, but there are no differences in the nature of the plaque components. In addition, the carotid and coronary arteries have many similarities in the physiology of vascular tone regulation that has an impact on plaque evolution. Myocardial blood perfusion is regulated by vasodilation of the epicardial coronary arteries in response to various stimuli such as NO, causing dynamic changes in coronary artery tone that may result in multiple changes in blood flow. In a similar manner, the carotid artery is not just a simple conduit supporting cerebral circulation; it exhibits vascular response properties in response to stimuli, which include changes in shear stress. Endothelial shear stress contributes to endothelial health and a favorable vessel wall transcriptome profile. Clinical studies have demonstrated that areas of low endothelial shear stress are associated with atherosclerosis development and high-risk plaque characteristics. Similarly, in the carotid arteries, lower wall shear stress is associated with plaque development and location. (Endothelial shear stress itself is a useful measurement, but not a substitute for plaque morphology.) It is important to recognize that technical challenges are different across beds (e.g., use of gating, vessel size, amount and nature of motion), but these effects are mitigated by scanning protocols, which result in approximately in-plane voxel sizes in the range of 0.5-0.75 mm, and through-plane resolution of the coronary (smaller vessels) is actually superior to, rather than inferior to, that of the carotid (where voxels are isotropic in the coronary, but not in the carotid and peripheral extremities).

[0092] The present disclosure is based on solid mathematical principles that respect the Nyquist-Shannon sampling theorem, and achieves effective resolution using conventionally acquired CTA in the same field of activity as IVUS VH. IVUS imaging has excellent spatial resolution for overall structure (i.e., lumen), but generally lacks the ability to characterize plaque components with high accuracy. Literature estimates that using a typical transducer in the range of 20-40MHz, the IVUS resolution is 70-200μm axially and 200-400μm laterally. IVUS VH is a method of spectral backscatter analysis that enables plaque composition analysis (and therefore measurement). For IVUS VH methods that use a large (e.g., 480μm) moving window in the axial direction, the relatively large size of this moving window (and therefore the accuracy of the composition analysis) is fundamentally limited by the bandwidth requirements of the spectral analysis. In the case where the IVUS VH image is displayed in a smaller moving window (e.g., 250μm), since each IVUS pixel is classified into a discrete category, this limits the accuracy of this analysis. 64-slice multi-detector CCTA scans have been described as being in the range of 300-400 μm resolution. Although this already brings CCTA resolution very close to that of IVUS VH, other factors unique to the analysis of the present invention are to be considered. Therefore, rather than discretely classifying CCTA pixels, the systems and methods of the present disclosure perform an iterative deblurring or restoration modeling step with subvoxel accuracy, such as using a triangular tessellated surface to represent the true surface of the lipid core. The lipid core area was in the range of 6.5-14.3 mm2 in 393 patients (corresponding to a radius of curvature of 1.4-2.1 mm). Using the chord length formula in which the chord spans a single voxel diagonally, this indicates that the upper limit of the error in the tessellated surface representation of the lipid core is 44 μm. There are also additional factors associated with the deblurring or restoration analysis that may cause an error of approximately half of the pixels with a total range of 194-244 μm in accuracy, which is generally equivalent to the accuracy of IVUS VH for measuring cap thickness.

[0093] The present disclosure is also innovative in dealing with the fundamental limitations of applying artificial intelligence and deep learning to the analysis of atherosclerosis imaging data. Conventional competing methods that lack validated and objective evidence are fundamentally limited in many ways. First, the use of arbitrary thresholds results in an inability to assess accuracy unless in a form of weak correlation with other markers that lack objective validation themselves. Second, this lack of objectivity increases the need for large amounts of data based on correlations in the environment. This places an unfeasible requirement on manual annotation of radiological images, which are themselves the objects of analysis (that is, as opposed to validation based on independent modalities). Third, due to the interpretability of the resulting models, these models must be presented to regulatory agencies such as the FDA as "black boxes" that lack scientifically rigorous descriptions of the mechanisms of action that can be linked to traditional biological hypothesis testing. Specifically, although CNNs have been shown to be excellent in performing many different computer vision tasks, they have significant drawbacks when applied to radiological data sets: 1) requiring large amounts of training and validation data sets, and 2) intermediates.

[0094] CNN computations typically do not represent any measurable properties, which makes regulatory approval difficult. To address these challenges, a pipeline approach consisting of stages and outputs that can be objectively verified at the biological level to feed the CNN is utilized. The present invention overcomes these disadvantages by using a pipeline consisting of one or more stages and subsequent smaller-scale convolutional neural networks, wherein the one or more stages are biomeasurable (i.e., can be objectively verified), and the subsequent smaller-scale convolutional neural networks process these verified biological properties to output the desired output conditions that can be said to be not based on subjective or qualitative "image features". These architectural capabilities reduce the disadvantages by increasing the efficiency of available training data and enabling intermediate steps to be objectively verified. The systems and methods of the present disclosure use CNNs for medical imaging to reduce the disadvantages by: 1) reducing the complexity of the visual task to a level acceptable when training CNNs with moderately sized data sets; and 2) producing intermediate outputs that are objectively verified and easy to interpret by users or regulators. Intermediate CNN computations often fail to represent any measurable properties, spatial context is often difficult to obtain in ML methods using feature extraction, and using raw datasets that do contain spatial context often lacks objective ground truth labels for the extracted features, whether processed using traditional machine learning or deep learning methods. Likewise, raw datasets contain variations that are "noisy" relative to the classification problem at hand, which is overcome in computer vision applications outside of medicine by having very large training sets of a certain size that are not typically available in the medical field, especially datasets annotated with ground truth.

[0095] The quantitative capabilities of the disclosed systems and methods make them ideal for analysis of more advanced imaging protocols such as early / delayed phase contrast, dual energy and multi-spectral techniques for tissue characterization studies.

[0096] While the system and method of the present disclosure has been particularly shown and described with reference to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] The foregoing will become apparent from the following more particular description of example embodiments as illustrated in the accompanying drawings in which like reference numerals refer to the same parts throughout the different views. The drawings are not necessarily drawn to scale, emphasis instead being placed upon illustrating embodiments of the present disclosure.

[0098] Figure 1 Depicted is a schematic diagram of an exemplary system for determining and characterizing a medical condition by implementing a hierarchical analysis framework in accordance with the present disclosure.

[0099] Figure 2 A resampling-based model building approach in accordance with the present disclosure that can be implemented by the systems and methods described herein is outlined.

[0100] Figure 3 Depicted are sample patient reports that may be output by the systems and methods described herein in accordance with the present disclosure.

[0101] Figure 4 Depicted are example segmentation stages of a multi-scale vessel wall analyte map in accordance with the present disclosure.

[0102] Figure 5 Depicted is an exemplary pixel-level probability mass function as a set of analyte probability vectors in accordance with the present disclosure.

[0103] Figure 6 Techniques for calculating putative analyte spots in accordance with the present disclosure are presented.

[0104] Figure 7 Depicted are normalized vessel wall coordinates for an exemplary vessel wall composition model in accordance with the present disclosure.

[0105] Figure 8 Depicted is an example edge between removed plaque and the outer vessel wall of a histological sample in accordance with the present disclosure.

[0106] Fig. 9 Some complex vascular topologies that can be interpreted using the techniques described herein in accordance with the present disclosure are shown.

[0107] Fig.10 Depicted are exemplary analyte spots representing normalized vessel wall coordinate distributions in accordance with the present disclosure.

[0108] Fig.11 Depicted is an exemplary distribution of blog descriptors in accordance with the present disclosure.

[0109] Fig.12 Depicted is an exemplary model for imaging data correlating between hidden ground truth states and observed states in accordance with the present disclosure.

[0110] Fig.13 Depicted is a diagram of an example Markov model / Viterbi algorithm for associating observed states with hidden states in an image model in accordance with the present disclosure.

[0111] Fig.14 Depicted is an example frequency distribution of the total number of spots per histology slide for a plurality of histology slides in accordance with the present disclosure.

[0112] Fig.15 Depicted is an exemplary implantation of a 1D Markov chain according to the present disclosure.

[0113] Fig.16 Depicted is an example first-order Markov chain for a text probability table according to the present disclosure.

[0114] Fig.17 Depicted is a conditional dependency of a first pixel based on its neighboring pixels in accordance with the present disclosure.

[0115] Fig.18 Additional exemplary hierarchical analysis frameworks according to the present disclosure are depicted.

[0116] Fig.19 Depicted is an example application for phenotyping purposes in guided vascular therapy according to the present disclosure. The depicted example uses the Stary plaque typing system adopted by the AHA as a basis, where the types determined in vivo are shown as color overlays.

[0117] Fig. 20 An example application for phenotyping lung cancer according to the present disclosure is depicted.

[0118] Fig.21Exemplary image preprocessing steps for deblurring or restoration according to the present disclosure are shown. Deblurring or restoration uses a patient-specific point spread determination algorithm to mitigate artifacts or image limitations caused by the image formation process that may reduce the ability to determine characteristics that predict phenotypes. The deblurred or restored (processed) image depicted (bottom) is derived from CT imaging of the plaque (top) and is the result of iteratively fitting a physical model of the scanner's point spread function using regularized assumptions about the true latent density of different regions of the image.

[0119] Fig. 22 Depicted are exemplary applications of datasets exhibiting enrichment for atherosclerotic plaque phenotypes in accordance with the present disclosure.

[0120] Fig.23 Tangential and radial directional variables are shown using an internal representation of unit phasors and phasor angles (shown encoded in gray scale) in accordance with the present disclosure, illustrating the use of normalized axes for tubular structures associated with blood vessels and other pathophysiologies associated with such structures (e.g., the gastrointestinal tract).

[0121] Fig.24 An exemplary overlay of annotations generated by a radiology analysis application from CTA (contour regions) on top of pathologist-generated annotations from histology (solid regions) in accordance with the present disclosure is shown.

[0122] Fig.25 Further steps of data enrichment in accordance with the present disclosure, specifically utilizing a normalized coordinate system to avoid non-correlated variations associated with wall thickness and radial representation, are presented. Specifically, the "donut shape" is "unfolded" while maintaining pathologist annotations.

[0123] Fig.26 A further step of data enrichment related to plaque phenotyping according to the present disclosure is represented. Working from the expanded formalism, luminal irregularities (eg due to ulceration or thrombus) and locally varying wall thickening are represented.

[0124] Fig. 27 Represents data-enriched imaging (in both unexpanded and expanded forms) containing intraplaque hemorrhage and / or other morphologies according to the present disclosure.

[0125] Fig.28 Validation results of algorithms trained and validated with different variations of data enrichment imaging according to the present disclosure are shown.

[0126] Fig.29 An example application of phenotyping lung lesions according to the present disclosure is provided.

[0127] Fig.30Examples of biological properties (including, for example, tissue characteristics, morphology, etc.) for phenotyping lung lesions according to the present disclosure are provided. Note that in example embodiments, pseudo colors may be represented as continuous values ​​rather than discrete values ​​for one or more of these biological properties.

[0128] Fig.31 A high-level view of an example method for user interaction with a computer-assisted phenotyping system according to the present disclosure is presented.

[0129] Fig.32 An example system architecture according to the present disclosure is presented.

[0130] Fig.33 Components of an example system architecture according to the present disclosure are presented.

[0131] Fig.34 is a block diagram of an example data analysis according to the present disclosure. Images of a patient are collected, the raw slice data is used in a set of algorithms to measure biological properties that can be objectively verified, and these biological properties are then formed into an enriched data set for feeding one of multiple CNNs, where the results are forward and backward propagated using a recursive CNN in this example to enforce constraints or generate continuous conditions (such as a monotonically decreasing blood flow reserve fraction from proximal to distal throughout the vascular tree, or a constant HRP value in a focal lesion or other constraints).

[0132] Fig.35 The causal relationships and available diagnoses of coronary ischemia in accordance with the present disclosure are presented.

[0133] Fig.36 Depicted are exemplary 3D segmentations of lumen, vessel wall, and plaque components (LRNC and calcification) of two patients presenting with chest pain and similar risk factors and degrees of stenosis according to the present disclosure. Left: 68-year-old male with NSTEMI at follow-up. Right: 65-year-old male without event at follow-up. The systems and methods of the present disclosure correctly predicted their respective outcomes.

[0134] Fig.37 Depicted are example histological processing steps for objectively verifying tissue composition according to the present disclosure.

[0135] Fig.38 is a schematic depiction of multiple cross sections that may be processed to provide dynamic analysis of a vessel tree across an entire scan in accordance with the present disclosure, illustrating the relationships between the cross sections and where the processing of one cross section depends on its neighbors. DETAILED DESCRIPTION

[0136] Systems and methods for analyzing pathology using quantitative imaging are presented herein. Advantageously, the systems and methods of the present disclosure utilize a hierarchical analysis framework that identifies biological properties / analytes from imaging data and quantifies the biological properties / analytes, and then identifies and characterizes one or more pathologies based on the quantified biological properties / analytes. This hierarchical approach of using imaging to examine the underlying biology as an intermediate for evaluating pathology provides many analytical and processing advantages relative to systems and methods that are configured to directly determine and characterize pathology based on raw imaging data without performing a validation step and / or without performing the advantageous processing described herein.

[0137] For example, one advantage is the ability to utilize training sets from non-radiological sources, such as histological information from tissue sample sources, in combination with or independent of training sets from radiological sources, to correlate radiological imaging features with biological properties / correlate analytes with pathology. For example, in some embodiments, histological information can be used in a training algorithm to identify and characterize one or more pathologies based on quantified biological properties / analytes. More specifically, biological properties / analytes that can be identified / quantified in non-radiological data (such as in invasively obtained histological data sets or obtainable by gene expression profiles) can also be identified in radiological data and the biological properties / analytes are quantified (which is advantageously non-invasive). Information from non-radiological sources can then be used, such as using histological information, gene expression profiles, or other clinically rich data sets, to correlate these biological properties / analytes with clinical findings about pathology. Such clinically relevant data sets can then serve as a training set or part of a training set for determining / tuning (e.g., using machine learning) algorithms that correlate biological properties / analytes with pathologies that have known relationships with clinical outcomes. These algorithms that correlate biological properties / analytes with pathologies obtained using non-radiological source training sets can then be applied to evaluate biological properties / analytes derived from radiological data. Thus, the systems and methods of the present disclosure can advantageously enable the use of radiological imaging (which can advantageously be cost-effective and non-invasive) to provide alternative measures for predicting clinical outcomes or guiding treatment.

[0138] It is noteworthy that in some cases, training data from non-radiological sources (such as histological information) can be more accurate / reliable than training data from radiological sources. In addition, in some embodiments, training data from non-radiological sources can be used to enhance training data from radiological sources. Therefore, since better data input may result in better data output, the disclosed hierarchical analysis framework advantageously improves the trainability and resulting reliability of the algorithms disclosed herein. As described above, a major advantage is that once trained, the systems and methods of the present disclosure can achieve the derivation of clinical information comparable to existing histological and other non-radiological diagnostic tests without undergoing invasive and / or expensive procedures.

[0139] Alternatively, in some embodiments, a training set of radiological sources (such as non-radiological imaging sources, e.g., histological sources and / or non-imaging sources) can be used in conjunction with or independently of a training set of radiological sources, e.g., when correlating image features with biological properties / analytes. For example, in some embodiments, one or more biological models can be extrapolated and fitted to correlate radiological data with non-radiological data. For example, histological information can be correlated with radiological information based on an underlying biological model. This correlation can enable trained identification of biological properties / analytes in radiological data utilizing non-radiological, e.g., histological information.

[0140] In some embodiments, data extracted from complementary modalities may be used to correlate, for example, image features with biological properties / analytes from blood panels, physical FFR, and / or other data sources.

[0141] In an example embodiment, imaging data extracted from one imaging modality can be used to extrapolate and fit one or more biological models, wherein the one imaging modality is correlated and / or fused with another imaging modality or non-imaging source (such as bloodwork). These biological models can advantageously be correlated across and between imaging data sets and non-imaging data sets based on the biological models. Therefore, these biological models can enable a hierarchical analysis framework to utilize data from one imaging modality with another imaging modality or with non-imaging sources to identify one or more biological properties / analytes / quantify or identify / characterize one or more medical conditions.

[0142] Another advantage of the hierarchical analysis framework disclosed herein is the ability to incorporate data from multiple data sources of the same or different types into a process for identifying and characterizing pathology based on imaging data. For example, in some embodiments, one or more non-imaging data sources can be used in combination with one or more imaging data sources to identify a set of biological properties / analytes and quantify the set of biological properties / analytes. Thus, in particular, the set of biological properties / analytes can include one or more biological properties / analytes identified and / or quantified based on one or more imaging data sources, one or more biological properties / analytes identified and / or quantified based on one or more non-imaging data sources, and / or one or more biological properties / analytes identified and / or quantified based on a combination of imaging and non-imaging data sources (note that for the purposes of the quantitative imaging systems and methods disclosed herein, the set of biological properties / analytes can generally include at least one or more biological properties / analytes identified and / or quantified based at least in part on imaging data). The ability to augment information from imaging data sources with information from other imaging and / or non-imaging data sources when identifying and quantifying a set of biological properties / analytes increases the robustness of the systems and methods presented herein and enables utilization of any and all relevant information in identifying and characterizing pathologies.

[0143] Yet another advantage of the hierarchical analysis framework relates to, for example, the ability to adjust / fine tune the data at each level before or after utilizing the data to evaluate subsequent levels (note that this can be an iterative process in some embodiments). For example, in some embodiments, information relating to a set of identified and quantified biological properties / analytes can be adjusted a posteriori (e.g., after preliminary identification and / or quantification thereof). Similarly, in some embodiments, information relating to a set of identified and characterized pathologies can be adjusted a posteriori (e.g., after preliminary identification and / or characterization thereof). These adjustments can be automatic or user-based and can be objective or subjective. The ability to adjust / fine tune the data at each level can advantageously improve data interpretability and reliability.

[0144] In an example embodiment, the adjustment can be based on contextual information, which can be used to update one or more probabilities that affect the determination or quantification of biological properties / analytes. In an example embodiment, the contextual information used to adjust the information related to a set of identified and quantified biological properties / analytes in a post hoc manner can include patient demographics, correlations between biological properties / analytes, or correlations between identified / characterized pathologies and biological properties / analytes. For example, in some cases, biological properties / analytes can be related in the sense that the identification / quantification of a first biological property / analyte can affect the probability related to the identification / quantification of a second biological property / analyte. In other cases, for example, the identification / characterization of a first pathology based on an initial group of identified / quantified biological properties / analytes may affect the probability related to the identification / quantification of biological properties / analytes in the initial group or even biological properties / analytes not in the first group. In other cases, pathologies can be related, for example, where the identification / characterization of a first pathology can affect the probability related to the identification / characterization of the first pathology. As described above, information related to identification and quantification of biological properties / analytes and / or information related to identification and characterization of pathologies may be updated in an iterative manner, for example until data converges or a threshold / benchmark is achieved or for a selected number of cycles.

[0145] Another advantage of the hierarchical analysis framework relates to the ability to provide information related to both pathology and basic biology to users, for example, physicians. This increased context can facilitate clinical diagnosis / assessment and evaluation / determination of the next step, for example, therapy / treatment options or other diagnosis. For example, systems and methods can be configured to determine that those biological parameters / analytes associated with the identification / quantification of one or more pathologies are the most uncertain / have the highest degree of uncertainty (for example, due to lack of data or conflicting data). In such cases, specific additional diagnosis can be recommended. Providing the user with the increased context of information related to both pathology and basic biology can further help the user evaluate / error check the various clinical conclusions and recommendations drawn by the analysis.

[0146] As used herein, the hierarchical analysis framework refers to an analysis framework in which one or more intermediate groups of data points are used as an intermediate processing layer or an initial group of data points and an intermediate transformation between the data points of the terminal group. This is similar to the concept of deep learning or hierarchical learning in which an algorithm is used to use multiple processing layers or to utilize multiple transformations, such as multiple nonlinear transformations, to model higher-level abstractions. Generally, the hierarchical analysis framework of the system and method of the present disclosure includes data points related to biological properties / analytes as intermediate processing layers or imaging data points and pathological data points. In examples and embodiments, multiple processing layers or multiple transformations (e.g., embodied by multi-level data points) for determining each of imaging information, basic biological information, and pathological information may be included. Although an example hierarchical analysis framework structure (e.g., with specific processing layers, transformations, and data points) is introduced herein, the system and method of the present disclosure are not limited to such embodiments. On the contrary, without departing from the scope and spirit of the present disclosure, any number of different types of analysis framework structures may be utilized.

[0147] In an example embodiment, the hierarchical analysis framework of the present application can be conceptualized as including a logical data layer that serves as an intermediate between an empirical data layer (including imaging data) and a result layer (including pathology information). However, the empirical data layer represents data from the direct source, and the logical data layer advantageously adds a degree of logic and reasoning that can extract this raw data into a set of useful analytes for the result layer in question. Thus, for example, empirical information from diagnosis, such as raw imaging information, can be advantageously extracted down to logical information related to biological features of a particular group, which is relevant for evaluating a selected pathology or pathology group (e.g., the pathology involves the imaged area of ​​the patient's body). In this way, the biological features / analytes of the present application can also be viewed as pathological symptoms / indicators.

[0148] The biological characteristics / analytes of the present application may sometimes be referred to as biomarkers in this article. Although the term "biological" or the prefix "biological" is used to characterize a biological characteristic or biomarker, this is only intended to indicate that the characteristic or marker has a certain degree of relevance relative to the patient's body. For example, a biological characteristic can be anatomical, morphological, compositional, functional, chemical, biochemical, physiological, histological, genetic or any number of other types of characteristics related to the patient's body. Example biological characteristics utilized by specific embodiments of the systems and methods of the present disclosure are disclosed herein (e.g., related to a specific anatomical region of the patient, such as the vascular system, respiratory system, organs such as the lungs, heart or kidneys, or other anatomical regions).

[0149] Although the example systems and methods of the present disclosure may be adapted for detecting, characterizing, and treating pathologies / diseases, application of the systems and methods of the present disclosure is not limited to pathologies / diseases, but rather may be more generally applicable to any clinically relevant medical condition of a patient including, for example, syndromes, disorders, trauma, allergic reactions, etc.

[0150] In an exemplary embodiment, the systems and methods of the present disclosure relate to computer-assisted phenotyping, such as by analyzing medical images using knowledge about biology to measure differences between disease types that have been determined through research to be indicative of a phenotype that can in turn predict an outcome. Thus, in some embodiments, characterizing a pathology can include determining a phenotype of the pathology, which can in turn determine a predicted outcome.

[0151] First reference Figure 1 , depicting a schematic diagram of an exemplary system 100. There are three underlying functions that may be provided by the system 100, represented as a trainer module 110, an analyzer module 120, and a cohort tool module 130. As depicted, the analyzer module 120 advantageously implements a hierarchical analysis framework that first utilizes a combination of (i) imaging features 122 from one or more acquired images 121A of a patient 50, and (ii) non-imaging input data 121B of the patient 50 to identify and quantify biological properties / analytes 130, and then identifies and characterizes one or more pathologies 124 (e.g., prognostic phenotypes) based on the quantified biological properties / analytes 123. Advantageously, the analyzer module 120 may operate independently of a ground truth or validation reference by implementing one or more pre-trained, e.g., machine learning, algorithms to derive its inferences.

[0152] In an example embodiment, the analyzer may include an algorithm for calculating imaging features 122 from acquired images 121A of the patient 50. Advantageously, some of the image features 122 may be calculated on a per-voxel basis, while other image features may be calculated on a region-of-interest basis. Example non-imaging inputs 121B that may be used with the acquired images 121A may include data from a laboratory system, patient-reported symptoms, or patient history.

[0153] As described above, image features 122 and non-imaging inputs can be used by the analyzer module 120 to calculate biological properties / analytes 123. Notably, biological properties / analytes are typically quantitative, objective properties (e.g., objectively verifiable rather than expressed as an impression or appearance) that can represent, for example, the presence and extent of markers (e.g., chemicals) or other measurements such as the structure, size, or anatomical characteristics of a region of interest. In an example embodiment, the quantified biological properties / analytes 123 can be displayed or exported for direct consumption by a user, e.g., a clinician, in addition to or independently of further processing by the analyzer module.

[0154] In an example embodiment, one or more of the quantified biological properties / analytes 123 may be used as input for determining a phenotype. A phenotype is typically defined in a disease-specific manner, independent of imaging, and typically extracted from an ex vivo pathophysiological sample with a documented relationship to an expected outcome. In an example embodiment, the analyzer module 120 may also provide a predicted outcome 125 for a determined phenotype.

[0155] It should be appreciated that example embodiments of the analyzer module 120 are further described herein with respect to specific embodiments that follow the general description of the system 100. In particular, specific imaging features, biological properties / analytes, and pathologies / phenotypes are described with respect to specific medical applications, such as with respect to the vascular system or with respect to the respiratory system.

[0156] Still reference Figure 1 , the cohort tools module 130 enables definition of cohorts of patients for group analysis thereof, e.g., based on a selected set of criteria relevant to the cohort study in question. An example cohort analysis may be for a group of patients enrolled in a clinical trial, e.g., where the patients are further grouped based on one or more arms of the trial, e.g., a treatment arm versus a control arm. Another type of cohort analysis may be for a set of subjects for which a baseline truth or reference exists, and this type of cohort may be further decomposed into a training set or "development" set and a test set or "holdout" set. A development set may be supported for training 112 algorithms and models within the analyzer module 120, and a holdout set may be supported for evaluating / validating 113 the performance of an algorithm or model within the analyzer module 120.

[0157] Continue to refer Figure 1, the trainer module 110 can be used for training 112 the algorithms and models in the analyzer module 120. Specifically, the trainer module 110 can rely on the reference truth 111 and / or the reference annotation 114 to facilitate deriving weights or models, for example, according to the machine learning paradigm established or by notifying the algorithm developer. In an exemplary embodiment, a highly adaptable, for example, classification and regression model that can reveal the complex relationship between the predictor and the response is adopted. However, its ability to adapt to the basic structure in the existing data can enable the model to find a pattern that is not reproducible for another sample of the subject. Adapting to the non-reproducible structure in the existing data is generally referred to as model overfitting. In order to avoid setting up an overfitting model, a systematic method can be applied, which prevents the model from finding a false structure, and enables the end user to have confidence that the final model will predict a new sample with a similar degree of accuracy with the data set that evaluates its model.

[0158] The continuous training set can be used to determine one or more optimal tuning parameters, and the test set can be used to estimate the predictive performance of the algorithm or model. The training set can be used to train each classifier in the classifier by randomized cross-validation. The data set can be repeatedly split into training and test sets and can be used to determine classification performance and model parameters. The splitting of the data set into training and test sets occurs using a hierarchical method or a maximum dissimilarity method. In an example embodiment, a resampling method (e.g., bootstrapping) can be used in the training set to obtain (i) the optimal parameter estimates and (ii) confidence intervals for the predictive performance of the model.

[0159] Figure 2 A resampling-based model building method 200 that can be utilized by the systems and methods of the present disclosure is outlined. First, at step 210, tuning parameter sets can be defined. Next, at step 220, a model is fit and a holdout sample is predicted for resampling each tuning parameter set data. At step 230, the resampled estimates are combined into a performance spectrum. Next, at step 240, the final tuning parameters can be determined. Finally, at step 250, the entire training set is refitted with the final tuning parameters. After each model has been tuned according to the training set, each model can be evaluated for predictive performance on a test set. The test set evaluation occurs once for each model to ensure that the model building process does not overfit the test set. For each model built, the best tuning parameter estimates, the resampled training set performance, and the test set performance can be reported. The values ​​of the model parameters over random splits are then compared to evaluate the model stability and robustness to the training data.

[0160] According to the systems and methods of the present disclosure, multiple models can be tuned for each of the biological properties / analytes (e.g., tissue types) represented in the baseline truth map. Model responses can include, for example, covariance-based techniques, non-covariance-based techniques, and tree-based models. Depending on its construction, the endpoint can have continuous responses and categorical responses, some of the techniques in the above categories are used for both categorical responses and continuous responses, while other techniques are specific to categorical responses or continuous responses. The best tuned parameter estimates, resampled training set performance, and test set performance can be reported for each model.

[0161] Table 1:

[0162]

[0163] Table 1 above provides a summary of some example functions of the analyzer module 120 of the system 100. That is, the analyzer module 120 can be configured to describe fields, such as to align multiple data streams across fields; to segment organs, vessels, lesions, and other application-specific objects; and / or to reformat / reconfigure anatomy for specific analysis. The analyzer module 120 can be further configured to describe targets, such as lesions, in the described fields. Describing targets can include, for example: aligning multiple data streams at a regional setting; performing fine-grained segmentation; measuring the size and / or other characteristics of related anatomical structures; and / or extracting entire target features (e.g., biological properties / analyte properties of the entire target area). In some embodiments, one or more sub-target areas can also be described, for example, a target area can be split into sub-targets (e.g., biological properties / analyte properties of sub-target areas) according to specific applications and calculations specific to sub-targets. The analyzer module 120 can also describe, for example, components or related features (such as compositions) in specific fields, targets, or sub-target areas. This can include segmenting or re-segmenting components / features, calculating the values ​​of the segmented components / features (e.g., biological properties / analyte characteristics of the components / features) and assigning probability maps to readings. The next pathology can be determined based on the biologically quantified properties / analytes, and the next pathology can be characterized, for example, by determining the phenotype and / or predicted results of the pathology. In some embodiments, the analyzer module 120 can be configured to compare data across multiple time points, for example, one or more of the biological components / analytes can be related to time-based quantification. In other embodiments, a wide scanning field can be utilized to evaluate multi-focal pathology, for example, based on the aggregate quantification of biological properties / analytes across multiple targets in the described field. Finally, based on the foregoing analysis, the analyzer module 120 can be configured to generate a patient report.

[0164] Sample patient report 300 depicted in Figure 3As shown, the sample patient report 300 can include quantification of biological parameters / analytes such as those related to structure 310 and composition 320, as well as data from non-imaging sources such as hemodynamics 330. The sample patient report can further include visualizations 340, such as 2D and / or 3D visualizations of imaging data and combined visualizations of non-imaging data such as hemodynamic data superimposed on the imaging data. Various analyses 350 can be displayed for evaluating biological parameters / analytes, the various analyses including, for example, visualizations of one or more models (e.g., decision tree models) for determining / characterizing pathology. Patient background and identification information can be further included. Therefore, the analyzer module 120 of the system 100 can advantageously provide comprehensive feedback to users, such as clinicians, for evaluating patients.

[0165] Advantageously, the systems and methods of the present disclosure can be adapted for specific applications. Example vascular and pulmonary applications are described in more detail in the following sections (although it should be understood that the specific applications described have general meaning and interoperability with respect to many other applications). Table 2 provides an overview of vascular and pulmonary related applications that utilize the hierarchical analysis framework as described herein.

[0166] Table 2:

[0167]

[0168] The following sections provide specific examples of quantitative biological properties / analytes that can be used by the systems and methods of the present disclosure with respect to vascular applications:

[0169] Anatomy: Vascular structural measurements, particularly those leading to the determination of % stenosis, have long been and remain the single most commonly used measurements in patient care. These measurements were originally limited to intraluminal measurements rather than wall measurements involving both the inner and outer surfaces of the vessel wall. However, all of the major noninvasive modalities, unlike X-ray angiography, can resolve the vessel wall and with resolution, an expansion of measurements that can be made. The categories are broad and the objects measured are of varying sizes, so generalizations should be made with caution. The primary consideration is the limitation of spatial sampling or resolution. However, by exploiting subtle changes in intensity levels due to partial volume effects, the minimum detectable change in wall thickness can be lower than the spatial sampling. Furthermore, the stated resolution usually refers to the grid size and the post-acquisition reconstructed field of view rather than the actual resolving power of the imaging protocol to determine the minimum feature size that can be resolved. Likewise, in-plane resolution may or may not be the same as across-plane resolution, and the size of a given feature as well as its proportions and shape will drive measurement accuracy. Last but not least, in some cases, classification conclusions are reached by applying thresholds to measurements, which can then be interpreted based on signal detection theory using the ability to optimize the trade-off between sensitivity and specificity (terms that do not otherwise refer to the measurement in the normal sense).

[0170] Tissue Characterization: Quantitative assessment of the individual component parts of an atherosclerotic plaque, including, for example, lipid-rich necrotic core (LRNC), fibrosis, intraplaque hemorrhage (IPH), magnetic permeability, and calcification, can provide critical information about the relative structural integrity of the plaque that can aid the physician's decision during medical or surgical therapy. From an imaging technology perspective, the ability to do this depends less on spatial resolution than on contrast resolution and tissue differentiation, which is made possible by differentiating tissues that respond differently to incident energy, thereby producing different received signals. Each imaging modality does this to some extent; terms such as "acoustic transparency" in ultrasound, the number of Hounsfield units in CT, and differential MR enhancement vary with various sequences such as (but not limited to) T1, T2, and T2*.

[0171] Dynamic tissue behavior (e.g., magnetic permeability): In addition to morphological features of the vessel wall / plaque, dynamic features are increasingly recognized as valuable quantitative indicators of vascular pathology. Dynamic sequences, where acquisition is performed at multiple closely spaced times (called phases), expand the library beyond spatially resolved values ​​t, which contain time-resolved values ​​that can be used for compartmental modeling or other techniques to determine the dynamic response of tissue to stimuli (such as but not limited to in- and out-washout of contrast agents). By using dynamic contrast-enhanced imaging using ultrasound or MR in the carotid arteries or delayed contrast enhancement in the coronary arteries, sensitive assessments of the relative magnetic permeability (e.g., Ktrans and Vp parameters from kinetic analysis) of the microvascular network of neovascularization within the plaque of interest can be determined. In addition, these dynamic series can also assist in the partitioning between increased vascular magnetic permeability and intraplaque hemorrhage.

[0172] Hemodynamics: The basic hemodynamic parameters of the circulation have a direct impact on vascular pathology. Blood pressure, blood flow velocity, and vessel wall shear stress can be measured by techniques ranging from very simple oscillometric methods to complex imaging analysis. Using the general principles of fluid dynamics, calculations of vessel wall shear stress in different regions of the wall can be determined. In a similar manner, MRI, with or without US, has been used to calculate wall shear stress (WSS) and to correlate the results with structural changes in the vessel of interest. In addition, the effects of antihypertensive drugs on hemodynamics have been studied in both the short and long term.

[0173] Thus, in example embodiments, key aspects of applying the systems and methods of the present disclosure in a vascular setting may include evaluating plaque structure and plaque composition. Evaluating plaque structure may advantageously include, for example, lumen measurements (which improve stenosis measurements by providing area rather than just diameter measurements) as well as wall measurements (e.g., wall thickness and vascular remodeling). Evaluating plaque composition may advantageously involve quantification of tissue properties (e.g., lipid core, fibrosis, calcification, magnetic permeability, etc.) rather than just "soft" or "hard" designations as commonly found in the prior art. Tables 3 and 4 below describe example structural calculations and tissue property calculations, respectively, that may be utilized by vascular applications of the systems and methods of the present disclosure.

[0174] Table 3: Structural calculations of vascular anatomy supported by vascular applications of the systems and methods disclosed herein

[0175]

[0176]

[0177] An example system related to evaluating a vascular system may advantageously include / adopt an algorithm for evaluating vascular structure. Thus, the system may employ, for example, a target / vessel segmentation / cross-section model for segmenting the underlying structure of the imaged vessel. Advantageously, a fast-marching competitive filter may be applied to individual vessel segments. The system may further be configured to handle vessel bifurcations. Image registration may be applied using Mattes mutual information (MR) or mean square error (CT) metrics, rigid versor transforms, LBFGSB optimizers, and the like. As described herein, vessel segmentation may advantageously include lumen segmentation. Initial lumen segmentation may utilize a filter of confidence connections (e.g., carotid arteries, vertebrae, femurs, and the like) to distinguish lumens. Lumen segmentation may utilize MR imaging (e.g., a combination of normalized (e.g., inverted) dark contrast images) or CT imaging (e.g., using registered pre-contrast, post-contrast CT, and 2D Gaussian distributions) to define a vessel-ness function. Various connected components may be analyzed, and thresholding may be applied. Vessel segmentation may further require outer wall segmentation (e.g., using minimum curvature (k2) flow to resolve lumen irregularities). In some embodiments, edge potential maps are calculated as outward-downward gradients in both contrast and non-contrast. In an example embodiment, outer wall segmentation can utilize a cumulative distribution function in a velocity function (e.g., merging a previous distribution of wall thickness, such as 1-2 adjacent levels) to allow for median thickness in the absence of any other edge information. In an example embodiment, a ferret diameter can be used for vessel characterization. In another embodiment, wall thickness can be calculated as the sum of the distance to the lumen plus the distance to the outer wall. In another embodiment, semantic segmentation can be used, using, for example, a CNN for lumen and / or wall segmentation.

[0178] The example system related to the evaluation of the vascular system can further advantageously analyze the composition of the blood vessels. For example, in some embodiments, the composition can be determined based on image intensity and other image features. In some embodiments, the lumen shape can be used, for example, related to determining thrombosis. Advantageously, the analyte spot model can be used to better analyze the composition of a specific sub-region of the blood vessel. The analyte spot is defined as a spatially continuous region in a 2D, 3D or 4D image of a class of biological analytes. The spot model can utilize an anatomically aligned coordinate system using contour lines, which are, for example, at a normalized radial distance from the lumen surface to the adventitia surface of the vessel wall. The model can advantageously identify one or more spots and analyze, for example, each spot position relative to the entire vascular structure and relative to other spots. In an example embodiment, a hybrid Bayesian / Markovian network can be used to model the relative position of the spots. The model can advantageously explain the image intensity observed at the pixel or voxel affected by the local neighborhood of the hidden analyte class node, thereby explaining the partial volume and the scanner point spread function (PSF). The model can further allow for dynamic description of analyte spot boundaries from the analyte probability map during inference of the analyzer module. This is a major difference from typical machine vision methods such as superpixel methods that precompute small regions to be analyzed but cannot dynamically adjust these regions. An iterative inference procedure that utilizes current estimates of both analyte probabilities and spot boundaries can be applied. In some embodiments, parameterized modeling assumptions or kernel density estimation methods can be used to achieve probability density estimates between sparse data used to train the model.

[0179] This article introduces a new model for classifying the composition of vascular plaque components, which eliminates the requirement for histology to radiology registration. This model still uses expert-annotated histology as a reference standard, but the training of the model does not require registration with radiological imaging. The multiscale model calculates the statistics of each continuous region of a given analyte type, which can be called a "spot". In a cross-section through a blood vessel, the wall is defined by two boundaries, the inner boundary of the lumen and the outer boundary of the vessel wall, thereby generating a donut-shaped shape in the cross-section. Within the donut-shaped wall region, there are a discrete number of spots (different from the default background class of normal wall tissue, which is not considered a spot). The number of spots is modeled as a discrete random variable. Then, a label of the analyte type is assigned to each spot, and various shape descriptors are calculated. In addition, the spots are considered to be paired. Finally, within each spot, each pixel can generate a radiological imaging intensity value, which is modeled as an independent and identically distributed (iid) sample from a continuous valuation distribution specific to each analyte type. Note that in this final step, the parameters of the imaging intensity distribution are not part of the training process.

[0180] A key feature of this model is that it takes into account the spatial relationship of analyte spots within the vessel and also between each other, recognizing that point-by-point image features (whether histologically and / or radiologically) are not the only source of information for expert determination of plaque composition. Although the model allows for training without explicit histological to radiological registration, it can also be applied to situations where the configuration is known. It is believed that statistically modeling the spatial layout of atherosclerotic plaque components to classify unseen plaques is a novel concept.

[0181] Example techniques for estimating vessel wall composition from CT or MR images are further described in the following sections. Specifically, the method can use a multi-scale Bayesian analysis model. The basic Bayesian formula is as follows:

[0182]

[0183] In the context of the present disclosure, the hypothesis may be based on a multi-scale vessel wall analyte map A, where observations are combed based on CT or MR image intensity information I.

[0184] like Figure 4As depicted, the multiscale vessel wall analyte map may advantageously include wall-level segmentation 410 (e.g., cross-sectional slices of the vessel), blob-level segmentation, and pixel-level segmentation 430 (e.g., based on individual image pixels). For example, A=(B,C) may be defined as a graph of vessel wall class labels (similar to a graph with vertices B and edges C), where B is a set of blobs (cross-sectional contiguous regions of non-background wall that share a label) and C is a set of blob pairs. B b can be defined as a generic single spot, where b∈[l..n B ] are the indices of all spots in A, B b a is a blob with label a. For statistical purposes, the individual blob descriptor operator D B {} is in a low-dimensional space. C c can be defined as a spot pair, where c∈[l..n B (n B -l) / 2] are the indices of all spot pairs in A. C c f,g is a blob pair with labels f and g. For statistical purposes, the blob pair descriptor operator Dc{} is in a low-dimensional space. A(x)=a can be defined as the class label of pixel x, where \a∈{'CALC','LRNC','TIBR','IPH','background'} (combined characteristics). In an exemplary embodiment, I(x) is the continuous estimated pixel intensity at pixel x. Within each blob, I(x) is modeled independently. Note that because the model is used to classify wall composition in 3D radiological images, the word "pixel" is used to refer genetically to both 2D pixels and 3D voxels

[0185] The characteristics of spot areas of similar composition / structure can advantageously provide insights into the disease process. Each slice of a blood vessel (e.g., a cross-sectional slice) can advantageously contain multiple spots. The relationship between the spots can be evaluated in a pairwise manner. The number of spots within a cross section is modeled as a discrete random variable and can also have quantifiable significance. At the segmented slice level, the relevant characteristics (e.g., biological properties / analytes) can include quantification of the total number of spots and / or multiple spots classified into a specific structure / composition, the relationship between the spots, for example, spatial relationships, such as closer to the inside. At the segmented spot level, the characteristics of each spot such as structural characteristics (e.g., size and shape) and combined characteristics, etc. can be evaluated as acting as biological properties / analytes. Finally, at the segmented pixel level, individual pixel-level analysis can be performed, for example, based on image intensity distribution.

[0186] Can be compared to Figure 4The multiscale vessel wall analyte map depicted in FIG. 1 applies probability mapping of characteristics. The probability mapping can advantageously establish a probability vector for each pixel, wherein a component of the vector is for the probability of each class of the analyte and one component is for the probability of background tissue. In an example embodiment, a set of probability vectors can represent mutually exclusive characteristics. Therefore, each set of probability vectors representing mutually exclusive characteristics will total 1. For example, in some embodiments, it may be known that a pixel should fall into one and only one component category (e.g., a single coordinate of a vessel cannot be both fibrous and calcified). It should be particularly noted that the probability mapping does not assume independence of analyte classes between pixels. This is because adjacent pixels or pixels within the same spot can generally have the same or similar characteristics. Therefore, the probability mapping account as described in more detail herein advantageously resolves the dependencies between pixels.

[0187] f(A=α) can be defined as the probability density of the map a. f(A) is the probability distribution function over all vessel walls. f(D B {B a}=β) is the probability density of the descriptor vector P with label a. f(D B {B a}) is the probability density function (pdf) of the blob descriptor with label a. For each value of a there exists a probability distribution function. f(B) = Πf(D B {B a})f(D c {C f,g}=γ) is the probability density of the pair of descriptor vectors γ with labels f and g. f(D c {C f,g}) is the probability density function (pdf) of the pairwise blob descriptors. For each ordered pair f, g, there exists a probability distribution function. Thus:

[0188] f(c)=Πf(D c {C a})

[0189] f(A)=f(B)f(C)=Πf(D b {B a})Πf(D c {C a})

[0190] P(A(x)=a) is the probability of pixel x having label a. P(A(x)) is the probability mass function (pmf) (prevalence) of the analyte. It can be viewed as a vector of probabilities at a specific pixel x, or a probability map for a specific class label value.

[0191] Note: f(A)=P(N)·f(C)·f(B)=P(N)·Πf(Cc )·Πf(B b )

[0192] f(C c =γ) is the probability density of the pairwise descriptor vector γ. f(C c ) is the probability density function (pdf) of the pairwise blob descriptors. f(B b =β) is the probability density of the descriptor vector β. f(B b ) is the probability density function (pdf) of the spot descriptor. P(A(x)=a) is the probability of a pixel x having label a. P(A(x)) is the probability mass function (pmf) of the analyte (the prevalence in a given image). It can be viewed as a vector of probabilities at a specific pixel x, or a spatial probability map of a specific analyte type. P(A(x)=a|I(x)=i) is the probability of an analyte of a given image intensity being the primary target of the calculation. P(I(x)=i|A(x)=a) is the distribution of image intensities for a given analyte.

[0193] Figure 5 An exemplary pixel-wise probability mass function as a set of analyte probability vectors is depicted. As described above, the following assumptions may inform the probability mass function: Completeness: In an exemplary embodiment, it may be assumed that sufficiently small pixels must fall into at least one of the analyte classes (including the general "background" class), and therefore the sum of the probabilities amounts to 1. Mutual exclusivity: It may be assumed that sufficiently small pixels belong to only one class of analytes; if there are combinations (i.e., spicate calcium in the context of LRNC), new combined classes may be created to maintain mutual exclusivity. Non-independence: It may be assumed that each pixel is highly dependent on its neighbors and the overall structure of A.

[0194] An alternative view of the analyte map is a spatial map of the probability of a given analyte. At any given point during inference, the analyte spot can be defined using the maximum full width at half maximum rule. Using this rule, for each local maximum number of the probability of the analyte, the region grows outward to a lower threshold of half the local maximum value. Note that this 50% value is an adjustable parameter. Spatial regularization of the spots can be performed by performing a certain curvature evolution on the probability map to keep the boundaries more realistic (smooth, with few topological holes). Note that different possible putative spots of different analyte classes can often have spatial overlaps, because until different possible putative spots collapse the probability, these spatial overlaps represent alternative hypotheses for the same pixel, and therefore the modifier is "putative".

[0195] When the iterative inference terminates, there are several options for the representation of the result. First, the continuous estimated probability map can be directly presented to the user in one of several forms including but not limited to surface mapping, contour mapping, or using an image fusion similar to the change of hue and saturation of the top of the CT that visualizes the PET value as CT. The second alternative is to collapse the probability map at each pixel by selecting a single analyte label for each pixel. This can be most directly performed by independently selecting the maximum a posteriori value at each pixel, thereby creating a visual classification map by assigning different colors to each analyte label and assigning full or partial opacity on the top of the radiology image. Under this second alternative, the label value can be distinguished by the priority probability based on each spot, rather than independently assigned. Therefore, at a given pixel, if a given pixel belongs to a spot of higher probability, the analyte probability of lower priority can be used for the label.

[0196] Figure 6 A technique for calculating pseudo putative analyte spots is presented. In an example embodiment, putative spots may have overlapping regions. Therefore, it may be advantageous to apply analytical techniques to segment pixels by putative spots. For the probability of a given analyte, the local maximum number of probabilities is determined. The maximum rule of the full width at half maximum can then be applied to determine discrete spots. In any given iteration of inference, the analyte spots can be defined using the full width at half maximum rule. The local maximum number is found, and then the region grows with a lower threshold of 0.5*max. (The value of 50% can be an adjustable parameter.) In some embodiments, spatial regularization of spots can also be applied, such as by performing a certain curvature evolution on the probability map in order to keep the boundaries smooth and avoid holes. Note that at this stage, different possible putative spots of different analyte classes can usually have spatial overlaps, because until the probability collapses, these represent alternative hypotheses. Therefore, an image-level analyte map is calculated, such as a collapse based on a probability map function. Note that this collapse can be determined based on a pixel-level analyte probability map, a putative spot, or a combination of the two. With respect to the pixel-level analyte probability map, collapse can be determined by selecting, for each pixel, the label with the maximum probability A(x): = arg maxa P(A(x) = a). This is similar to the implementation of the Viterbi algorithm. Basically, the highest probability of each mutually exclusive set of probability vectors is locked (for example, analyte priorities break possible ties). All other probabilities in the set can then be set to zero. In some embodiments, when collapsing data at the pixel level, the probabilities of adjacent pixels / regions can be considered. With respect to putative spot-level collapse, overlapping putative spots can be distinguished. In some embodiments, prioritization can be based on spot probability density f(D1{A a b=d1). This may affect the analysis of blob-level characteristics since higher probability blobs may change the shape of overlapping lower probability blobs. In an example embodiment, the full range of probabilities may be maintained rather than collapsing the data.

[0197] In order to model the relative spatial positioning of spots within the vessel wall, an appropriate coordinate system can be selected to provide rotational invariance, translational invariance, and scale invariance between different images. These invariances are important for the model because, under the assumption that the atherosclerotic process is similar across different vascular beds, it allows the ability to train on one type of vessel (e.g., the carotid artery where endarterectomy samples are readily available) and apply the model to other vascular beds (e.g., the coronary where plaque samples are generally not readily available). For tubular objects, the natural coordinate system follows the vessel centerline, where the distance along the centerline provides the longitudinal coordinate, and each plane perpendicular to the centerline has the polar coordinates of the radial distance and angle. However, due to the variability of the geometry of the vessel wall, particularly in sick patients who may be intended for analysis, an improved coordinate system may be utilized. The longitudinal distance is calculated in a manner such that each 3D radiology image pixel is given a value, either along the centerline or along an interpolated vertical plane. For a given patch, the proximal and distal planes perpendicular to the centerline are each used to create an unsigned distance map on the original image grid, which is denoted as P(x) and D(x), where x represents the 3D coordinates, respectively. The distance map l(x) = P(x) / (P(x) + D(x)) represents the relative distance along the patch, with a value of 0 at the proximal plane and a value of 1 at the distal plane. The direction of the l axis is determined by ▽l(x).

[0198] Because the geometry of the wall can be significantly non-circular, the radial distance can be defined based on the shortest distance to the inner luminal surface and the shortest distance to the outer adventitial surface. Expert annotations of histological images contain regions that define the lumen and vessels (defined as the union of the lumen and vessel wall). Signed distance functions can be created for each of these L(x) and V(x), respectively. The convention is that the interior of these regions is negative, so that in the wall, L is positive and V is negative. The relative radial distance is calculated as r(x)=L(x) / (L(x)-V(x)). It has a value of 0 at the luminal surface and a value of 1 at the adventitial surface. The direction of the r axis is determined by ▽r(x).

[0199] Due to the geometry of the non-circular wall, the normalized tangential distance can be defined to lie along the isoline r (or isoline 1 if processing in 3D). The direction of the t axis is determined by ▽r×▽l. The convention is that histological sections are assumed to be viewed from the proximal to distal direction, so positive 1 points into the image. Note that unlike the others, t does not have a natural origin because it wraps around the vessel to itself. Therefore, the origin of this coordinate can be defined differently for each spot relative to the centroid of the spot.

[0200] Another wall coordinate used is normalized wall thickness. In a sense, this is a surrogate marker for disease progression. Thicker walls are assumed to be due to more severe disease. It is assumed that the statistical relationship of the analyte changes with more severe disease. Absolute wall thickness is easily calculated as w abs (x) = L(x) - V(x). To normalize it to the range [0-1], the maximum possible wall thickness can be determined when the lumen is close to zero size and completely eccentric and close to the outer surface. In this case, the maximum diameter is the maximum Ferrier diameter D of the vessel max Therefore, the relative wall thickness is calculated as w(x) = w abs (x) / D max .

[0201] The extent to which the aforementioned coordinates may or may not be used in the model depends in part on the amount of available training data. When training data is limited, several options are available. Different segments of each plaque are treated, and the relative longitudinal distances can be ignored as if they were from the same statistical distribution. It has been observed that plaque composition varies along the longitudinal axis, with the middle having a more severe plaque appearance. However, in contrast to parameterizing the distribution by l(x), this dimension can be collapsed. Similarly, relative wall thickness can also be collapsed. It has been observed that certain analytes appear in the "shoulder" region of the plaque, where w(x) will have an intermediate value. However, this dimension can also be collapsed until sufficient training data is available.

[0202] As described above, a vessel wall composition model may be utilized as an initial assumption (eg, at the previous P(A)). Figure 7Normalized vessel wall coordinates of an exemplary vessel wall composition model are depicted. In the depicted model, l is the relative longitudinal distance from proximal to distal along the vessel target, which can be calculated, for example, in normalized intervals [0,1]. The longitudinal distance can be calculated using 2 fast-travel propagations starting from the proximal plane and the distal plane to calculate unsigned distance fields P and D, where 1=P / (P+D). The l axial direction is ▽l. As depicted, r is the normalized radial distance, which can also be calculated in normalized intervals [0,1] from the luminal surface to the adventitial surface. Therefore, r=L / (L+(-V)), where L is the luminal signed distance field (SDF) and V is the vessel SDF. The r axial direction is ▽r. Finally, t is the normalized tangential distance, which can be calculated, for example, in normalized intervals [-0.5,0.5]. Note that in example embodiments, there may not be a meaningful origin for the entire wall, only for individual analyte spots (thus, the t origin may be at the spot centroid). The tangent distance is calculated along the contour curves of 1 and r. The t axis direction is ▽r×▽l.

[0203] Fig. 9 Some complex vascular topologies are shown, which can be resolved using the techniques described herein. Specifically, when processing CT or MR in 3D, different branches can be advantageously analyzed separately so that the relationship between analyte spots in the separate branches is appropriately ignored. Therefore, if the segmented view (cross-sectional slice) contains more than one lumen, this can be accounted for by performing a watershed transform on r so that the wall is split into domains belonging to each lumen, and each domain can then be considered / analyzed separately.

[0204] As mentioned above, many of the coordinates and probability measures described herein can be expressed using a normalized scale, thereby maintaining scale invariance, such as between vessels of different sizes. Therefore, under the assumption that the disease process is similar and proportional for vessels of different calibers, the proposed model can be advantageously independent of absolute vessel size.

[0205] In some embodiments, the model can be configured to characterize concentric plaques versus eccentric plaques. Notably, all thicknesses normalized close to 1 can indicate highly eccentric locations. In further embodiments, inward plaque characterization versus outward plaque characterization can be implemented. Notably, histological information about this property is hampered by deformation. Therefore, in some embodiments, CT and training data can be used to establish an algorithm for determining inward plaque characterization versus outward plaque characterization.

[0206] As described above, in example embodiments, non-imaging data such as histology data may be utilized as a training set for building algorithms that link image features to biological properties / analytes. However, there are some differences between the types of data that need to be addressed when ensuring proper correlation. For example, the following differences between histology and imaging may affect proper correlation: Carotid endarterectomy (CEA) leaves behind the adventitia and some media in the analysis of CT or MR images of patients where an outer adventitial surface is assumed to be found. (See e.g. Figure 8 , the figure depicts the edge between the removed plaque of the histological sample relative to the outer vessel wall). It is worth noting that the scientific literature shows uncertainty as to whether calcification will occur in the adventitia. The following techniques can be used to explain this difference. Histology can expand outward, for example, based on the assumption that there is little analyte left in the wall. Alternatively, image segmentation can erode inward, for example based on the knowledge of typical or specific edges that are retained. For example, an average edge can be utilized. In some embodiments, the average edge can be normalized as a percentage of the total diameter of the vessel. In other embodiments, histology can be used to mask the imaging (e.g., based on an alignment standard overlay). In such embodiments, one or more transformations may need to be applied to the histological data to match the appropriate alignment. Ultimately, in some embodiments, the difference can be ignored (which is equivalent to a uniform scaling of the removed plaque to the entire wall). Although this may induce some small errors, it can be speculated that the remaining wall may be thinner than the plaque in CEA patients.

[0207] There may also be longitudinal differences between the histological data (e.g., training set) and the imaging data as represented by the vessel wall composition model. In example embodiments, longitudinal distances can be explicitly modeled / correlated. Thus, for example, the histological slice number (e.g., AG) can be used to roughly determine the location within the resected portion of the plaque. However, this approach limits the analysis relative to other slices for which there is no corresponding histological data. Therefore, alternatively, in some embodiments, all histological slices can be considered to arise from the same distribution. In example embodiments, some limited regularization can still be employed along the longitudinal direction.

[0208] As described above, normalized wall thickness is in some sense an imperfect surrogate for disease progression. In particular, it is assumed that thicker walls are due to more severe disease, for example based on the assumption that the statistical relationship of the analyte changes with more severe disease. Normalized wall thickness can be calculated as follows: The absolute wall thickness T can be determined a (in mm), for example, calculated as T a =L+(-V), where L is the lumen SDF, V is the vessel SDF, and D max is the maximum Ferret diameter of the blood vessel (in mm). Then we can calculate the diameter based on T = T a / D max The relative wall thickness T is calculated, for example in the interval [0, 1], where 1 indicates the thickest part of the tubule lumen, which indicates that the plaque is completely eccentric. In an example embodiment, the probability can be adjusted based on the wall thickness, for example so that the distribution of analyte spots will depend on the wall thickness. This can advantageously model differences in analyte composition during disease progression.

[0209] Fig.10 An exemplary analyte spot represented in the context of a normalized vessel wall coordinate distribution is depicted. Specifically, the origin of t is located at the spot centroid. The (r, t) coordinates are random vectors where the position / shape is completely represented by the joint distribution within the point. This can be simplified by considering the marginal distribution (since the radial and tangential shape properties appear to be relatively independent). The marginal distribution can be calculated as a projection along r and t (note that l and T coordinates can also be considered). Notably, the marginal distribution in the radial direction can advantageously represent / characterize plaque growth in concentric layers (e.g., medial layer, adventitia layer, and intima layer). Similarly, the marginal distribution in the tangential direction can advantageously represent growth factors that can indicate the stage of the disease. In an exemplary embodiment, an analyte spot descriptor can be calculated based on the marginal distribution. For example, on can perform low-order statistics on the marginal distribution (or use a histogram or fit a parametric probability distribution function).

[0210] In an example embodiment, the following analyte spot descriptors may be used, for example, to capture the location, shape, or other structural characteristics of individual spots:

[0211] - Position in normalized vessel coordinates

[0212] ● Mainly relative to r

[0213] - For example, to facilitate differentiation between shallow / deep calcifications

[0214] ●t direction is ignored; [optional model l direction]

[0215] - The extent of the normalized vessel coordinates

[0216] ● The word "size" is intentionally avoided, which implies an absolute measurement, whereas degree is a normalized value

[0217] - Asymmetry, which is used to indicate the degree of asymmetry in the distribution

[0218] ● Clinical significance is unclear, but it can help normalize shapes for incredibly asymmetrical shapes

[0219] - Alignment, which is used to represent the definition of parallel tissue layers

[0220] The analyte spots appear to be well preserved within radial layers (contours of r), so this will help select similar image processed shapes

[0221] -The wall thickness where the spot is located

[0222] Thick (eg, heavy) plaques are assumed to have different statistics than thin plaques

[0223] In some embodiments, paired blob descriptors may also be used. For example:

[0224] -Relative position

[0225] ●For example, if the fibrosis is on the luminal side of the LRNC

[0226] - Relative degree

[0227] ● For example, how thick / wide is the fibrosis relative to the LRNC

[0228] -surround

[0229] How close is the projection of one edge to the middle of another edge

[0230] ●For example, napkin ring sign or fibrosis around LRNC

[0231] -Relative wall thickness

[0232] ● Used to indicate the degree of "shoulder" (the shoulder will be relatively less thick than the main body of the central plaque)

[0233] Note that higher order interactions (e.g., between three blobs or between two blobs and another feature) may also be implemented. However, decreasing returns and training limits may be considered.

[0234] Here is an example quantification of the blob descriptor:

[0235] ● Individual spot descriptors

[0236]

[0237] Paired Blob Descriptors

[0238]

[0239] It is worth noting that a set of descriptors (e.g., 8-12 descriptors) forms a finite shape space that the blobs reside in. The distribution of a group of blobs can then be viewed as a distribution in this finite space. Fig.11An exemplary distribution of blog descriptors is depicted. In an example embodiment, the distribution of blob descriptors can be computed over the entire training set. In some embodiments, lower order statistics (assuming independence) can be utilized on individual blob descriptors, e.g., position: E[α r ],Var[α r ]. In other embodiments, a multidimensional Gaussian (mean vector + covariance matrix) analysis may be used to model the descriptors (eg, where independence is not assumed). In further embodiments, if the distribution is non-normal, it may be modeled using density estimation techniques.

[0240] As mentioned above, the number of spots per cross section (or the number of each class) can also be modeled, for example, η does not take into account the analyte class, and η i The number of each analyte species was counted. Fig.14 The frequency distribution of the total number of spots per histology slide is depicted. Toxicity regression was used as an excess. Note that Fig.14 The analytical graph of depicts the number N of spots per cross section regardless of analyte class (the number of spots for each analyte type is represented by B).

[0241] To summarize the above sections, in an exemplary embodiment, the entire vessel wall composition model may include the following:

[0242] Analytes per pixel before pmf

[0243] -P(A(x)=a i )=ρ i

[0244] ● Individual spot descriptors

[0245] -B1=(α r ,β r ,β t ,γ r ,γ t ,δ r ,δ t ,τ t )

[0246] -B1~N(μ 1, Σ1)

[0247] Paired Blob Descriptors

[0248] -C2=(α rr ,α tt ,β rr ,β tt ,ε rr ,ε tt ,τ TT )

[0249] -C2~N(μ2,Σ2)

[0250] The number of blobs

[0251] -η~Poisson(λ η )

[0252] in:

[0253] P(A(x)=a l )=ρ l

[0254] f(A b )=f(B1 b )

[0255]

[0256] As described above, the imaging model can be used as the likelihood of the Bayesian analysis model (e.g., P(I\A)). The maximum likelihood estimate can then be determined. In an example embodiment, this can be performed by considering each independent pixel (e.g., without considering the prior probability of the structure in the model). The estimated analyte map is usually only smoothed because the image is smooth (this is why pre-smoothing is usually not performed). Independent pixel-by-pixel analysis can be performed, for example, at least until the point where the scanner PSF is considered. Imaging models are used to explain imperfect imaging data. For example, imaging a small component of a plaque adds independent noise to the top of the pixel value. In addition, local volume effects and scanner PSFs are well known for their application to small objects. Therefore, given a model (e.g., a level set representation of an analyte region), it is simple and fast to simulate CT by utilizing Gaussian blurring of PSF. The imaging model described herein can also be applied to determine (or estimate) the distribution of the true (unblurred) density of different analytes. It is worth noting that this cannot come from typical imaging studies because these will have blurred image intensities. In some embodiments, a wide variance can be used to represent uncertainty. Alternatively, the distribution parameters can be optimized based on a training set, but the objective function will have to be based on downstream readouts (of the analyte region), for example, unless aligned histology data are available. Fig.12 Depicted is an exemplary model for imaging data (e.g., associating hidden (classified) states (A(x)) with observed (continuous) states (I(x)), taking into account random (e.g., analyte density distribution (H(A(x)) and deterministic (e.g., scanner blur *G(x))) noise factors). θ are the parameters of H (proportion of each analyte and HU mean / variance). For N different analyte classes assuming a normal distribution, θ = (τ1, μ1, σ1, …, τ N ,μ N ,σ N). Note that θ is patient-specific and will be estimated in an expectation-maximization (EM) manner, for example, where the analyte labels are latent variables and the images are observed data.

[0257] ●E-step: Determine the membership probability given the current parameters

[0258] M-step: maximize the likelihood of the parameters given the membership probabilities

[0259] Fig.13 A diagram depicting an example Markov model / Viterbi algorithm for relating observed states to hidden states in an image model. Specifically, the diagram depicts observed states (grey) (observed image intensity, I(x)) and hidden states (white) (pure analyte intensity, H(A(x))) that can be modeled with an empirical histogram or with a Gaussian or boxcar probability distribution function. The PSF of the imaging system is modeled as a Gaussian G(x). Thus,

[0260] I(x)=G(x)*H(A(x))

[0261] It should be noted that a Viterbi-like algorithm can be applied here, but the convolution will replace the emission probability H which can be modeled as Gaussian or uniform.

[0262] As mentioned above, one part of the inference procedure is based on expectation maximization (EM). In a typical application of EM, data points are modeled as belonging to one of several unknown classes. Each data point has a feature vector and for each class, this feature vector can be modeled using a parameter distribution represented by a mean vector and a covariance matrix, such as a multidimensional Gaussian. In the context of the model presented herein, a straightforward EM implementation would work as follows:

[0263]

[0264]

[0265] The main problem with this simple model is that it does not encode any higher order structure to the pixels. There is no prior probability associated with a more realistic pixel arrangement. Only tau determines the proportion of the analyte class. Therefore, once the tau variable can be used to insert in the spot prior probability model, especially in the step of updating the membership probability.

[0266] Therefore, a modified Bayesian inference procedure can be applied with much more complex Bayesian priors. In the basic EM implementation, there is no true prior distribution. The variable tau represents the prior relative proportion of each class, but even so, this variable is unspecified and estimated in the inference procedure. Therefore, there is no prior belief about the class distribution in the basic EM model. In the model, the model prior is represented by a multi-scale analyte model. Tau becomes a function of position (and other variables), not just the global proportion.

[0267]

[0268]

[0269] The membership probability function is defined as follows:

[0270]

[0271] The inference algorithm is as follows. At each step of the iteration, the membership probability map is initialized to zero so that the probability of all classes is zero. Then for all possible model configurations, the membership probability map can be added as follows:

[0272]

[0273] Finally, the probability vector can be normalized at each pixel in the member probability map to restore the integrity assumption. Advantageously, all model configurations can be iterated. This is done by considering the value of N from 0 to a relatively low value, such as 9, at which point it is observed that very few segments have so many spots. For each value of N, different putative spot configurations can be checked. The putative spots can be thresholded to a small number (N) based on the individual spot probabilities. Then, all permutations of the permutations of the N spots are considered. Therefore, all spot configurations in the most likely spot configuration can be considered simultaneously, and each model is weighted by its prior probability. This procedure is obviously an approximate inference scheme because the entire space of multi-scale model configurations may not be considered. However, it can be assumed that a good approximation is achieved by considering the most likely (in terms of both N and spots). This procedure also assumes that the weighted average of the most likely configuration provides a good estimate at each individual pixel. Another alternative is to perform a constrained search of the model configuration and select the highest likelihood model as a MAP (maximum a posteriori) estimate.

[0274] Additional exemplary statistical models (e.g., posterior P(A\I)) are also described herein. In CT angiography, the following information is available:

[0275] Strength

[0276] - CT Hounsfield units or MR intensity

[0277] - Possible other imaging features

[0278] ● Location relative to anatomy

[0279] - where in the patch, the pixels are

[0280] ●Neighboring pixels

[0281] - For example, for smoothing contours via level sets

[0282] The posterior probability can be calculated as:

[0283] P(A|I)∞P(I|A)·P(A)

[0284] Therefore, the following image information may affect the probability of the analyte Ai(x)

[0285] I(x) is the observed image intensity (may be a vector)

[0286] T(x) is the observed relative wall thickness according to image segmentation

[0287] F(x) is the CT image feature

[0288] S(x) is the characteristic of the vessel wall shape (e.g., lumen convexity)

[0289] In some embodiments, a method similar to Metropolis-Hastings may be utilized. In other embodiments, a maximum a posteriori method may be applied.

[0290] The following are example algorithmic possibilities for statistical analysis models. In some embodiments, the model can utilize confidence propagation (AKA maximum sum, maximum product, and product messaging). Therefore, for example, a Viterbi (HMM) type of method can be utilized, for example, where the hidden state is the analyte distribution A and the observed state is the image intensity I. This method can advantageously find that MAP estimation can be a selection mechanism P(A|I). In some embodiments, a soft output Viterbi algorithm (SOVA) can be utilized. Note that the reliability of each decision can be indicated by the difference between the selected (survivor) path and the discarded path. Therefore, this can indicate the reliability of each pixel analyte classification. In another example embodiment, a forward / backward Baum-Welch (HMM) method can be utilized. For example, the most likely state can be calculated at any point in time, but the most likely sequence cannot be calculated (see Viterbi).

[0291] Another possible technique is the Metropolis-Hastings (MCMC) method, for example, where one approach is to repeatedly sample and weight A by the likelihood and prior. In some embodiments, a simple MRF version for sampling can be utilized. Note that it may be particularly advantageous to sample the posterior directly. In an example embodiment, a per-pixel histogram of the analyte class may be established.

[0292] Other algorithmic possibilities include applying Gibbs samplers, variational Bayes (similar to EM), mean field approximation, Kalman filters, or other techniques.

[0293] As described above, in some embodiments, an expectation-maximization (EM) a posteriori approach may be utilized. Under this approach, the observed data X are imaged values, the unknown parameters θ are due to the analyte map (but do not include analyte probabilities), and the latent variable Z is the analyte probability vector. A key feature of this approach is that it iterates between estimated class membership (Z) and model parameters (θ) due to their dependence on each other. However, since the analyte map separates out the analyte probabilities, the approach may be modified so that the current class membership does not have to affect the model parameters (since these are learned in the training step). Thus, EM essentially learns the model parameters as it iterates through the current data. Advantageously, an exemplary embodiment of the EM approach iteratively computes the maximum likelihood, but assumes a flat prior.

[0294] Also provided herein is a technique for representing longitudinal covariance. Due to the wide spacing of histological sections (e.g., 4 mm), sampling may not be able to capture the longitudinal changes in the analyte truthfully. However, 3D image analysis is usually performed, and hypothetically, there are some real longitudinal covariances. The problem is that histological information is not usually provided for longitudinal covariance. Nevertheless, the exemplary statistical model disclosed herein can reflect slow changes in the longitudinal direction.

[0295] In some embodiments, a Markov model / chain may be applied. Fig.15 Depicted is an exemplary implantation of a 1D Markov chain for text / DNA. Conventionally, when applied to images in an MRF, the Markov chain is usually as low order as possible. However, higher order chains may be advantageous due to conditional independence (Markov properties). Otherwise the data may be too disordered to be valuable. This is demonstrated by a 1D sampling of an exemplary Markov chain as applied to text:

[0296] ●Uniform sampling output:

[0297] -earryjnv anr jakroyvnbqkrxtgashqtzifzstqaqwgktlfgidmxxaxmmhzmgbyamjgxnlyattvc rwpsszwfhimovkvgknlgddou nmytnxpvdescbg k syfdhwqdrjjmcovoyodzkcofmlycehpcqpuflje xkcykcwbdaifculiluyqerxfwlmpvtlyqkv

[0298] ●0th order Markov chain output:

[0299] -ooyusdii eltgotoroo tih ohnnattti gyagditghreay nm roefnnasos r naaeuuecocrrfca ayasel s yba anoropnn laeo piileo hssiod idlif beeghecebnnioouhuehinely neiis cnitcwasohsooglipyocp h trog 1

[0300] ●1st order Markov chain output:

[0301] -icke inginatenc blof ade and jalorghe y at helmin by hem owery fa stsin rdn cke st wanks hinioro e orin en s ar whes ore jot j whede chrveblan ted sesourethegebe inaberens s ichathfle watt o

[0302] ●2nd order Markov chain output:

[0303] -he ton th as my caroodif flows an the er ity thayertione wil ha mothenre re crearaquichow mushing whe so mosing bloack abeenem used shesigembs inglis day p wer wharonthe graiddid wor thad k

[0304] ● Output of the 3rd order Markov chain:

[0305] -es in angull o shoppinjust stees ther a kercourats allech is hotetemal liked be weavybecause in coy mrs hand room him rolio und ceran in thathe mound a dishine when what tobitcho way forgot p

[0306] Fig.16 Depicted is an example first-order Markov chain for a text probability table. Note that the size of such a table is exponential in order:

[0307] D = order of the Markov chain

[0308] N = number of letters

[0309] Size = N D

[0310] Therefore, higher orders lead to dimensionality problems. Advantageously, histological samples have very high resolution. However, since histological samples are not statistically independent, this may lead to overfitting, as will be described in more detail later. In general, the higher the conditional dependencies modeled, the more predictive the model will be.

[0311] In an example embodiment, a 2D Markov Random Field (MRF) may be used for the pixel values ​​instead of a ID sequence as for the letters. Fig.17 Depicted is the conditional dependency of a first pixel (black) based on its neighboring pixels (grey). In an example embodiment, the clique can exploit symmetry to reduce the number of dependencies by half. In some embodiments, the value of a pixel can be a simple image intensity or can be a probability value for a classification problem. There are problems with the use of typical MRFs. Conventional MRFs are almost always limited to the nearest neighbor pixels that provide conditional dependencies, which greatly reduces the specificity of the probability space represented, and for very general, typically only black / white spots,

[0312] purpose of segmentation / filtering; very short range dependencies. However, although pixels are highly discrete, a blob that misses just one pixel and falls into the next can completely change the probability distribution. Therefore, the true image structure is much more continuous than what is usually solved using MRFs.

[0313] For this reason, the systems and methods of the present disclosure may advantageously utilize inference procedures, such as, for example, a Bayesian type rule for a posteriori: likelihood × prior (P(A / I) α P(I / A) × P(A)). Using a crossword type analogy, the inference procedures implemented by the systems and methods of the present application are a bit like trying to do OCR on a crossword puzzle based on a noisy scan. Knowledge (even imperfect knowledge of a few squares) can also help inform the unknown squares in the crossword puzzle. By considering both the vertical and horizontal directions simultaneously, more efficiency is improved. In an example embodiment, the inference procedure can be heuristic. For example, one can initialize with no priors, and then solve the simpler problems first, which can provide clues about the harder problems to be solved later. Thus, the relatively easy detection of biological properties such as concentrated calcium can inform the existence of other harder problems for detecting analytes such as lipids. Each step of the inference procedure can narrow the probability distribution of the unresolved pixels.

[0314] As mentioned above, in order to obtain available data, a higher order Markov chain is preferred. The disadvantage of using a higher order Markov method is that there may not be enough data to inform the inference procedure. In an example embodiment, this problem can be solved by using a density estimation method such as a Parzen window or using a kriging technique.

[0315] To form an inference procedure, one may initialize with the unconditional prior probability of the analyte, and then use the highest level of evidence to begin narrowing the probabilities. For example, in some embodiments, an uncertainty width may be associated with each analyte probability estimate. In other embodiments, close to 1 / N may represent such uncertainty.

[0316] It should be noted that the term "Markov" is used loosely in this article, as the proposed Markov implementation is not memoryless but explicitly attempts to model long-range (spatial) dependencies.

[0317] Because CT resolution is low compared to histology and plaque anatomy, it may be preferred in some embodiments to utilize a continuous space (time) Markov model rather than a discrete space (time). This works well with a level set representation of the probability map, as it naturally works well with sub-pixel interpolation. The discrete analyte states make the model a discrete space model. However, if a model represents continuous probabilities rather than analyte presence / absence, it becomes a continuous space model.

[0318] Turning to pulmonary-based applications, Table 4 below depicts exemplary biological properties / analytes that may be utilized with respect to a hierarchical analysis framework for such applications.

[0319] Table 5: Biologically objective measurands supported by pulmonary-based applications

[0320]

[0321]

[0322] Specifically, the system can be configured to detect lung lesions. Therefore, the exemplary system can be configured for whole lung segmentation. In some embodiments, this may involve using minimum curvature evolution to solve the problem of near-pleural lesions. In some embodiments, the system can implement lung component analysis (blood vessels, fissures, bronchi, lesions, etc.). Advantageously, Hessian filters can be used to promote lung component analysis. In some embodiments, lung component analysis can further include pleural involvement, for example, depending on the geometry of the fissures. In other embodiments, attachment to anatomical structures can also be considered. In addition to lung component analysis, separate analysis of ground glass state and solid state can also be applied. This can include determining geometric features such as volume, diameter, sphericity, image features such as density and mass, and fractal analysis.

[0323] Fractal analysis can be used to infer scale growth patterns. In order to perform fractal analysis on very small areas of interest, the method adaptively modifies the support for the convolution kernel to limit it to the area of ​​interest (i.e., lung nodules). For the purpose of fractal analysis, intersecting blood vessels / bronchial and non-lesion features can be masked out. This is done by applying an IIR Gaussian filter on the masked local neighborhood and normalizing using an IIR blurred binary mask. In some embodiments, fractal analysis may further include determining porosity (variance based on the local mean). This can be applied relative to lung lesions, sub-portions of lesions. In an example embodiment, an IIR Gaussian filter or a circular neighborhood may be applied. In some embodiments, variance may be calculated using IIR. The average (AVL) of the local variance may also be calculated, for example, as applied to lung lesions. Similarly, the variance of the local variance may be calculated.

[0324] In an example embodiment, both lesion structure and composition can be calculated. Advantageously, calculating lesion structure can utilize the full volume of the thin segment, thereby improving upon calculating size measurement changes. Measurements such as subsolid and ground glass opacity (GGO) volumes can also be determined as part of assessing lesion structure. Turning to lesion composition, tissue characteristics such as consolidation, trauma, proximity, and perfusion can be calculated, thereby, for example, reducing false positive rates relative to conventional analysis.

[0325] Reference now Fig.18 , depicts another exemplary hierarchical analysis framework 1800 of the system of the present disclosure. Fig.18 can be understood as Figure 1 Detailed description of hierarchical inference system, which is described in more detail relative to the exemplary intermediate processing layer of the hierarchical inference system. Advantageously, hierarchical inference still flows from imaging data 1810 to basic biological information 1820, to clinical disease 1800. However, it is worth noting that framework 1800 includes multiple levels of data points for processing imaging data in order to determine biological properties / analytes. At preprocessing stage 1812, physical parameters, registration transformations, and regional segmentation can be determined. Then this preprocessed imaging information can be used to extract the imaging features 1814 of the next level of data points, such as intensity features, shape, texture, time characteristics, etc. The extracted image features can then be used at stage 1816 to fit one or more biological models to the imaged anatomy. Example models can include Bayesian / Markov network lesion substructures, fractal growth models, or other models as described herein. Biological models can advantageously serve as a bridge for relating imaging features to basic biological properties / analytes at stage 1822. Exemplary biological properties / analytes include anatomical structures, tissue composition, biological functions, gene expression correlations, etc. Finally, at stage 1832, the biological properties / analytes may be utilized to determine clinical findings related to the pathology, including, for example, disease subtype, prognosis, decision support, etc.

[0326] Fig.19 It is an example application for the purpose of phenotyping in guided vascular therapy, which uses the Stary plaque classification system adopted by the AHA as a basis, in which the types determined in vivo are shown as color overlays. The left figure shows an example of labeling according to the possible dynamic behavior of plaque lesions based on their physical properties, and the right figure shows an example of using the classification results to guide patient treatment. The example mapping is ['I', 'II', 'III', 'IV', 'V', 'VI', 'VII', 'VIII'], resulting in class_map = [subclinical, subclinical, subclinical, subclinical, unstable, unstable, stable, stable]. This method is not related to Stary, for example, the Virmani system ["calcified nodule", "CTO", "FA", "FCP", "healing plaque rupture", "PIT", "IPH", "rupture", "TCFA", "ULC"] has been used with class-map = [stable, stable, stable, stable, stable, stable, unstable, unstable, unstable, unstable], and other classification systems can obtain similar high performance. In example embodiments, the systems and methods of the present disclosure may incorporate completely different classification systems, may change class diagrams, or other changes. For FFR phenotypes, values ​​such as normal or abnormal may be used and / or quantities may be used to facilitate comparison with, for example, physical FFR.

[0327] Fig. 20is an example of a different disease, such as lung cancer. In this example, the subtype of the mass is determined in order to guide the most likely beneficial treatment to the patient based on the distinct phenotype.

[0328] It is expected that CNNs may perform better than read vector classification because CNNs contain filters that extract spatial context that is not included in (only) analyte area measurements. Despite the reduced training effort, it may be practical to use CNNs because

[0329] 1) There are relatively few classes corresponding to significantly different treatment alternatives (rather than being completely granular as can be done in research assays requiring ex vivo tissue), e.g., three phenotypes for classification problems, three risk levels for outcome prediction / risk stratification problems, so the problem is generally easier.

[0330] 2) Performing the substantial part of image interpretation by generating segmentations and presenting a simplified but fairly enriched dataset to the classifier, such as by processing analyte regions as pseudo-color regions via level sets or other algorithmic classes. Measurable pipeline stages reduce the dimensionality of the data (reducing the complexity of the problem the CNN must solve) while also providing verifiable intermediate values ​​that can increase confidence in the entire pipeline.

[0331] 3) Reformatting the data using a normalized coordinate system removes noise variation due to variables that have no substantial effect on the classification (e.g., vessel size in the case of plaque phenotyping).

[0332] To test this idea, a pipeline consisting of three stages was built:

[0333] 1) Semantic segmentation, which is used to identify which areas of the biomass fall into certain classes

[0334] 2) spatial unfolding, which is used to transform the vein / artery cross section into a rectangle, and

[0335] 3) A trained CNN that reads the annotated rectangle and identifies which class it belongs to (stable or unstable).

[0336] Without loss of generality, the example systems and methods described herein can apply spatial unfolding (e.g., training and testing a CNN with (unfolded dataset) and without (donut-shaped dataset) spatial unfolding). Observe that unfolding improves validation accuracy

[0337] Semantic Segmentation and Spatial Unfolding:

[0338] First, the image volume is preprocessed. This may include target initialization, normalization, and other preprocessing, such as deblurring or restoration to form a region of interest containing the physiological target to be phenotyped. The region is a volume consisting of cross-sections through the volume. The body part is determined automatically, or explicitly provided by the user. Targets of body parts that are tubular in nature are accompanied by a centerline. The centerline may have branches when present. Branches may be marked automatically or by the user. A generalization of the centerline concept may represent anatomy that is not tubular but benefits from some structural orientation, such as the region of a tumor. In any case, the centroid of each cross-section in the volume is determined. For tubular structures, the centroid will be the center of the lumen of the passage, for example, a blood vessel. For lesions, the centroid will be the center of the mass of the tumor.

[0339] Fig.21 Represents an exemplary image preprocessing step, in this case deblurring or restoration using a patient-specific point spread determination algorithm to mitigate artifacts or image limitations caused by the image formation process that may reduce the ability to determine characteristics that predict a phenotype. The figure illustrates a portion of an analysis applied to a radiological analysis of plaque based on CT. Shown here is a deblurred or restored image that is the result of iteratively fitting a physical model of the scanner's point spread function using regularized assumptions about the true latent density of different regions of the image. This figure is included to illustrate that various image processing operations can be performed to assist in being able to perform the quantification step, and is by no means an indication that this method is necessary for the specific invention in the present disclosure, but rather as an example of steps that can be taken to improve overall performance.

[0340] The (optionally deblurred or restored) images are represented in a Cartesian dataset where x is used to represent how far from the centroid, y represents the rotation θ, and z represents a cross section. Each branch or region will form one of these Cartesian sets. When multiple sets are used, "null" values ​​are used for overlapping regions, that is, each physical voxel will be present only once in the entire set, in such a way that they fit together geometrically. Each dataset will be paired with another dataset with sub-regions labeled with objectively verifiable tissue composition (see, e.g., Fig.36 ). Example labels for vascular tissue may be lumen, calcification, LRNC, etc. Example labels for lesions may be necrosis, neovascularization, etc. These labels may be objectively verified, for example, by histology (see, e.g., Fig.37 The paired dataset will be used as input for the training step of building a convolutional neural network. Two layers of analysis are supported, one on the individual cross-section layer, optionally where the output varies continuously across adjacent cross-sections, and a second layer on the volumetric layer (where the individual cross-sections can be viewed as still frames, and the vessel tree traversal can be viewed similar to a movie).

[0341] Example CNN design:

[0342] AlexNet is a CNN, which was completed in the ImageNet LargeScale Visual Recognition Challenge in 2012. The first 5 errors achieved by the network were 15.3%. AlexNet was designed by a supervisory group, which was composed of Alex Krizhevsky, Geoffrey Hinton and Ilya Sutskever of the University of Toronto (U Toronto) at that time. AlexNet was trained from the beginning to classify a set of independent images (not used in the training and verification steps during network training). For the expanded data, the AlexNet type network of 400 × 200 pixel input was used, and the donut-shaped network was the AlexNet type of 280 × 280 pixel input (resolution is roughly the same but aspect ratio is different). All values ​​in the convolution filter value were initialized, where weights were extracted from the AlexNet trained on the ImageNet data set. Although the ImageNet data set is a natural image data set, this is only used as an effective method for weight initialization. Once training begins, all weights are adjusted to better fit the new task. Most of the training plans were extracted directly from the open source AlexNet implementation, but some adjustments were required. Specifically, the base learning rate was reduced to 0.001 (solver.prototxt) and the batch size was reduced to 32 (train_val.prototxt) for both the AlexNet donut network and the AlexNet unfolded network. All models were trained to 10,000 iterations and compared to snapshots after training until only 2,000 iterations of training. Although a more in-depth study of overfitting can be performed, it is generally found that the reduction in both training and validation errors is between 2k and 10k iterations.

[0343] Alternative features (prefixes) can include:

[0344] ●ResNet-https: / / arxiv.org / abs / 1512.03385

[0345] ●GoogLeNet-https: / / www.cs.unc.edu / ~wliu / papers / GoogLeNet.pdf

[0346] ●ResNext-https: / / arxiv.org / abs / 1611.05431

[0347] ●ShuffleNet V2-https: / / arxiv.org / abs / 1807.11164

[0348] ●MobileNet V2-https: / / arxiv.org / abs / 1801.04381

[0349] Runtime optimizations such as frame-to-frame redundancy between cross sections (sometimes called "temporal" redundancy, but in this case a form of intra-cross-section redundancy) can be exploited to save computation (e.g., http: / / arxiv.org / abs / 1803.06312). Many optimizations for training or inference can be implemented.

[0350] In an example test implementation, AlexNet was trained to classify an independent set of images between two categories of clinical significance, e.g., "unstable" plaques versus "stable" plaques, based on histological ground truth plaque types V and VI, the latter comprising plaque types VII and VIII that adhere to the industry de facto standard plaque classification nomenclature accepted by the American Heart Association (AHA) and on Virmani's related but unique classification system.

[0351] Without loss of generality, in the examples shown, both total accuracy and confusion matrix are used to evaluate performance. This formalism is based on the concept of calculating four probabilities in a binary classification system: true positive, true negative, false positive and false negative. In example embodiments, other outcome variables can be used, however, for example, sensitivity and specificity can be used as outcome variables or F1 score (harmonic sum average of accuracy and sensitivity). Alternatively, AUC characteristics can be calculated for binary classifiers. In addition, the classifier does not need to be binary-based. For example, in some embodiments, the classifier can be sorted based on more than two possible states.

[0352] Dataset enhancement:

[0353] Physician annotated data is expensive, so it is desirable to artificially augment medical datasets (e.g., for training and / or validation). Two different augmentation techniques are used in the example embodiments described herein. The donut is randomly flipped horizontally, and rotated to random angles between 0 and 360. The resulting rotated donut is then cropped to the extent that the donut exists, and then padded with black pixels to fill the image to have a square aspect ratio. The result is then scaled to 280×280 size and saved as PNG.

[0354] The unrolled dataset is augmented by random horizontal flips and then “rolled” by a random number of pixels ranging from 0 to the width of the image. The result is then scaled to 400×200 size and saved as PNG.

[0355] Both datasets were augmented by a factor of 15, meaning the total number of images after augmentation is 15 times the original number. Class normalization was implemented, meaning the final dataset has roughly the same number of images belonging to each class. This is important because the original number of images for each class may be different, thereby biasing the classifier towards the class that has a larger number of images in the training set.

[0356] Without loss of generality, each radiologist performing the annotation can use any number of tissue types.

[0357] Fig. 22 An exemplary application for demonstrating various aspects of the present invention is shown, in this case, a classification for atherosclerotic plaque phenotype classification (e.g., using specific subject data). Different colors represent different tissue analyte types, with dark gray showing otherwise normal walls. The figure shows the results of the baseline truth annotation of tissue properties, which indicates the plaque phenotype and how it exists in the spatial context of a cross-section extracted orthogonal to the axis of the vessel. It also shows a coordinate system that has been developed to provide a common basis for the analysis of a large number of histological cross-sections. Grid lines are added to show the coordinate system (tangent and radial distances) and superimposed on top of the color-coded pathologist annotations. An important aspect is that this type of data set can be effectively used for deep learning methods because it uses relatively simple pseudo-color images instead of higher resolution full images to simplify the information, but without losing spatial context, for example, with a formal representation of such representations as the "napkin ring sign", near-luminal calcium, thin (or thick) cap (the interval between the LRNC and the lumen).

[0358] Fig.23 Tangential and radial directional variables are shown using unit phasors and internal representations of the phasor angles shown here in gray scale encoding, which illustrates the use of normalized axes for tubular structures associated with blood vessels and other pathophysiology associated with such structures (e.g., gastrointestinal tract). Note that the vertical bars from black to white are purely arbitrary boundaries due to gray scale encoding, and the normalized radial distance has a value of 0 at the luminal boundary and a value of 1 at the outer boundary.

[0359] Fig.24An exemplary overlay of annotations generated by a radiology analysis application based on CTA (unfilled color outlines) on annotations generated by a pathologist based on histology (solid color areas) is shown. An exemplary aspect of the systems and methods presented herein is that contour forms of in vivo non-invasive imaging can be used with classification schemes to non-invasively determine phenotypes, where the classifier is trained on known reference truth. Specifically, filling in the contours shown unfilled in this figure (so as not to obscure the relationship to the ex vivo annotations for this specific segment, which are provided to show correspondence) creates input data for the classifier.

[0360] Fig.25 Shown specifically, using a normalized coordinate system to avoid data enrichment of non-correlated changes associated with wall thickness and radial representation. Specifically, the "donut-like" is "unfolded" while maintaining pathologist annotations. The left figure shows the pathological region annotations of the histological sections of the plaque after deformation to convert the cut "C" shape back to the in vivo "O" shape of the complete vessel wall. The horizontal axis is the tangential direction around the wall. The vertical axis is the normalized radial direction (the bottom is the luminal surface and the top is the outer surface). Also note that the finer granularity of the pathologist's annotations has collapsed to match the granularity intended for extraction by in vivo radiological analysis applications (e.g., LRNC, CALC, IPH). The right figure shows the radiological analysis application annotations that can be comparable expanded. The axis and color are the same as the pathologist's annotations.

[0361] Fig.26 The following refinements are shown in relation to the plaque phenotyping example. Working from the expanded formalism, luminal irregularities (e.g., caused by ulceration or thrombus) and localized variable wall thickening are presented. The light grey at the bottom represents the lumen (added in this step to represent luminal surface irregularities), and the black used in the previous step has now been replaced with a dark grey to represent variable wall thickening. The black now completely represents the area outside the wall.

[0362] Fig. 27 Additional examples are shown to include, for example, intraplaque hemorrhage and / or other morphological aspects as needed (eg, using specific subject data). The left image shows a donut-shaped representation, and the right image is expanded, showing the luminal surface and localized wall thickening.

[0363] The example CNNs tested include CNNs based on the AlexNet and Inception frameworks.

[0364] AlexNet results:

[0365] In the example embodiment tested, the convolution filter values ​​were initialized with weights extracted from AlexNet trained on the ImageNet (referenced here) dataset. While the ImageNet dataset is a natural image dataset, this is only used as an effective method for weight initialization. Once training begins, all weights are adjusted to better fit the new task.

[0366] Most of the training schedule was extracted directly from the open source AlexNet implementation, but some adjustments were required. Specifically, the base learning rate was reduced to 0.001 (solver.prototxt) and the batch size was reduced to 32 (train_val.prototxt) for both the alexnet donut network and the alexnet unfolded network.

[0367] All models were trained to 10,000 iterations and compared to a snapshot after training until only 2,000 iterations. Although a deeper study of overfitting could be done, it was generally found that the reduction in both training and validation error was between 2k and 10k iterations.

[0368] The new AlexNet network model was trained from scratch on 4 (four) different combinations of ground truth results from two leading pathologists, two different ways of processing the images (see above), and on both unfolded and donut-shaped images. The results are presented in Fig.28 With class normalization enabled, each dataset variation had its training data augmented by 15 times. The networks were trained on this augmented data and then tested on the corresponding unaugmented validation data corresponding to the variation. For the expanded data, an AlexNet-type network with 400×200 pixel input was used, and the donut-shaped network was an AlexNet-type with 280×280 pixel input (roughly the same resolution but different aspect ratio). Note that in the test embodiment, the sizes of the regular layers as well as the fully connected layers changed. Therefore, the network in the AlexNet test embodiment can be described as a five convolutional layer, three fully connected layer network. Without loss of generality, here are some high-level conclusions based on these results:

[0369] 1) With the exception of the WN_RV dataset, it does seem that the unwrapped data is easier to analyze as it receives higher validation accuracy across the board

[0370] 2) The non-normalized data proved to be more representative, as expected.

[0371] 3) Regarding the WN-RV dataset, the initial idea was to pool the WN and RV truth data to see the compatibility of the typing systems and the extent to which the sets could be merged. In doing so, significant differences were observed in the WN and RV data. The original purpose of the WN_RV experiment was to pool the training data from multiple pathologists to see if the information would help with efficacy. Instead, degradation rather than improvement was observed. This was determined to be due to changes in the color scheme, which prevented this pooling of the data. Therefore, normalizing the color scheme to enable pooling could be considered.

[0372] Exemplary Alternative Network: Inception:

[0373] Transfer learning retraining of the Inception v3 CNN started with the August 8, 2016 version of the network uploaded for public use on the TensorFlow website. The training was performed for 10,000 steps. The training and validation sets were normalized in terms of the number of images by image augmentation, so that both subsets had the same number of annotated images in total. All other network parameters were taken to their default values.

[0374] A pre-trained CNN can be used to classify imaging features using the output of the last convolutional layer, which in the case of the Google Inception v3 CNN is a digital tensor of dimension 2048 × 2. Then, an SVM classifier is trained to recognize objects. This process is usually performed on the Inception model after a transfer learning and fine-tuning step, in which a model originally trained on the ImageNet 2014 dataset has its last softmax layer removed and retrained to recognize new categories of images.

[0375] Alternative Embodiments:

[0376] Fig.29 An alternative embodiment for phenotyping potentially cancerous lung lesions is provided. The leftmost figure indicates the outline of the segmented lesion, which has been preprocessed to separate it into solid and semi-solid areas ("ground glass") sub-regions. The middle figure indicates its location in the lung, and the rightmost figure shows it with a pseudo-color overlay. In this case, the 3-dimensional nature of the lesion may be considered significant, so, in contrast to processing 2D cross-sections alone, techniques such as video interpretation from computer vision can be applied to the classifier input data set. In fact, processing multiple cross-sections sequentially, as if arranged in a "movie" sequence along the centerline, can generalize these methods for tubular structures.

[0377] Another generalization is that the pseudocolor is not selected from a discrete palette, but has a continuous value at the pixel or voxel location. Using the lung example, Fig.30 A set of features is shown, which is sometimes described as so-called "radiomics" features that can be calculated for each voxel. This set of values ​​can exist in any number of pre-processed stacks and fed into a phenotype classifier.

[0378] Other alternative embodiments include using, for example, change data collected from multiple time points, rather than (only) data from a single time point. For example, if the amount or property of a negative cell type increases, this may be referred to as a "progressor" phenotype, while a "regressor" phenotype is for a decrease. A regressor may be, for example, due to a response to a drug. Alternatively, if, for example, the rate of change of LRNC is rapid, this may suggest a different phenotype. Extension of the examples to the use of delta values ​​or rates of change will be apparent to those skilled in the art.

[0379] As a further alternative, non-spatial information such as derived from other assays (e.g., lab results) or demographic / risk factors or other measurements extracted from radiological images can be fed to the final layer of the CNN to combine the spatial information with the non-spatial information. Likewise, localization information can be determined, such as by inferring the full 3D coordinates at the imaging, using pressure lines and readings at one or more certain locations along the vessel from references such as bifurcations or foramina.

[0380] Although these examples focus on phenotype classification, similar methods can be applied to the problem of outcome prediction as an additional embodiment of the present invention.

[0381] Example Implementation:

[0382] The systems and methods of the present disclosure may advantageously include a pipeline consisting of multiple stages. Fig.34Additional example implementations of the hierarchical analysis framework are provided. Biological properties are identified and quantified by semantic segmentation to identify biological properties represented digitally and in spatially unfolded enriched datasets (in an example application, lipid-rich necrotic cores, cap thickness, stenosis, dilation, remodeling ratios, tortuosity (e.g., inlet and outlet angles), calcifications, IPH, and / or ulcers), individually or in combination, to convert vein / artery cross-sections into rectangles and then into medical conditions (e.g., fractional flow reserve FFR causing ischemia, high-risk phenotype HRP, and / or time to risk stratification events (TTE)). The enriched datasets are read using one or more trained CNNs to identify and characterize conditions. Images of patients are collected, raw slice data is used in a set of algorithms to measure biological properties that can be objectively verified, and these biological properties are then converted into The physical properties are formed into an enriched data set for feeding one of multiple CNNs, in this example where the results are forward and backward propagated using a recursive CNN to enforce constraints or generate continuous conditions (such as a monotonically decreasing blood flow reserve fraction from proximal to distal throughout the vessel tree, or a constant HRP value in a focal lesion or other constraints). The ground truth data for HRP can exist as a pathologist determines the type of plaque that has been determined ex vivo at a given cross-section. The ground truth data for FFR can come from a physical pressure line, with one or more measurements and network training, the position proximal to a given measurement along the centerline is constrained to be greater than or equal to the measurement, the position distal is constrained to be less than or equal to the measurement, and when two measurements on the same centerline are known, the value between the two measurements will be constrained to be within an interval.

[0383] These properties and / or conditions may be determined at a given time point and / or may vary across time (longitudinal). Without loss of generality, other embodiments that perform similar steps in plaque phenotyping or other applications will be embodiments of the present invention.

[0384] In example embodiments, the biological properties may include one or more of the following:

[0385] Angiogenesis

[0386] New blood vessel formation

[0387] Inflammation

[0388] Calcification

[0389] Lipid deposition

[0390] Necrosis

[0391] Bleeding

[0392] Ulcers

[0393] ● Rigidity

[0394] Density

[0395] Stenosis

[0396] Expansion

[0397] ●Remodeling ratio

[0398] ●Tortuosity

[0399] Blood flow (eg, the flow of blood through blood vessels)

[0400] Pressure (eg, the pressure of blood in a channel or the pressure of one tissue against another)

[0401] Cell type (eg, macrophage)

[0402] ● Cell alignment (eg, of smooth muscle cells)

[0403] Shear stress (eg, shear stress of blood in the channel)

[0404] The analysis may comprise determining one or more of the amount, extent and / or character of each of the aforementioned biological properties.

[0405] Conditions that can be determined based on biological properties may include one or more of the following:

[0406] Perfusion / ischemia (if limited) (eg, perfusion / ischemia of brain or cardiac tissue)

[0407] Perfusion / infarction (eg, resection) (eg, perfusion / infarction of brain or cardiac tissue)

[0408] Oxygenation

[0409] Metabolism

[0410] Flow reserve (perfusion capacity), such as FFR (+) vs (-) and / or serial number

[0411] Malignant tumors

[0412] ●Violation

[0413] ●High-risk plaques, such as HRP(+) vs (-) and / or labeled phenotypes

[0414] Risk stratification (either as probability of event or time to event) (eg, explicit mention of MACCE)

[0415] Truth-based forms of verification can include the following:

[0416] Biopsy

[0417] ●Expert organization annotation of the tissue being resected (eg, endarterectomy or autopsy)

[0418] Expert phenotypic annotation of resected tissue (eg, endarterectomy or autopsy)

[0419] ●Physical pressure line

[0420] Other imaging modalities

[0421] ●Physiological monitoring (eg, ECG, SaO2, etc)

[0422] ●Genomic and / or proteomic and / or metabolomic and / or transcriptomic assays

[0423] Clinical outcomes

[0424] The analysis can be both at a given point in time and can be longitudinal (i.e., over time).

[0425] Example system architecture:

[0426] Fig.31 A high-level view of users and other systems interacting with the analytics platform according to the systems and methods of the present disclosure is presented. Stakeholders for this view include system administrators, support technicians with interoperability, security, failover and disaster recovery, and regulatory issues.

[0427] The platform can be deployed in two main configurations: on-premise or on a remote server ( Fig.32 ). The platform deployment can be a standalone configuration (top left), an on-premise server configuration (bottom left), or a remote server configuration (right). The on-premise deployment configuration can have two sub-configurations; desktop only or rack-mount. In a remote configuration, the platform can be deployed on a HIPAA compliant data center. The client accesses the API server over a secure HTTP connection. The client can be a desktop or tablet browser. No hardware is deployed at the customer site other than a computer running a web browser. The deployed server can be on a public cloud or on an extension of the customer's private network using a VPN.

[0428] The exemplary embodiment includes a client and a server. For example, Fig.33 The client is shown as a C++ application and the server as a Python application. These components interact using HTML 5.0, CSS 5.0, and JavaScript. Where possible, open standards are used for the interface, including but not limited to; HTTP(S), REST, DICOM, SPARQL, and JSON. Third-party libraries are also used as shown in this view, which shows the main pieces of the technology stack. Many variations and different approaches will be appreciated by those skilled in the art.

[0429] Various embodiments of the above systems and methods may be implemented in digital electronic circuitry, in computer hardware, firmware, and / or software. The embodiments may be implemented as a computer program product (i.e., a computer program tangibly embodied in an information carrier). The embodiments may be, for example, in a machine-readable storage device and / or in a propagated signal for execution by a data processing device, or to control the operation of a data processing device. The embodiments may be, for example, a programmable processor, a computer, and / or multiple computers.

[0430] The computer program may be written in any form of programming language, including compiled and / or interpreted languages, and the computer program may be deployed in any form, including as a stand-alone program or subroutine, component, and / or other unit suitable for a computing environment. The computer program may be deployed to be executed on one computer or on multiple computers at one site.

[0431] The method steps may be performed by one or more programmable processors executing a computer program to perform the functions of the invention by operating on input data and generating output. The method steps may also be performed by, and the apparatus may be implemented as, a special purpose logic circuit system. The circuit system may be, for example, an FPGA (field programmable gate array) and / or an ASIC (application specific integrated circuit). Modules, subroutines, and software agents may refer to a portion of a computer program, processor, special purpose circuit system, software, and / or hardware that implements the described functions.

[0432] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer may include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, operably coupled to receive data from one or more mass storage devices and / or transfer data to one or more mass storage devices.

[0433] Data transfer and instructions may also take place over a communications network. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices. The information carrier may be, for example, an EPROM, EEPROM, flash memory device, magnetic disk, internal hard disk, removable disk, magneto-optical disk, CD-ROM and / or DVD-ROM disk. The processor and memory may be supplemented by and / or incorporated in special purpose logic circuitry.

[0434] In order to provide interaction with the user, the above-mentioned techniques can be implemented on a computer with a display device. The display device can be, for example, a cathode ray tube (CRT) and / or a liquid crystal display (LCD) monitor. The interaction with the user can be, for example, a keyboard and a pointing device (e.g., a mouse or trackball) through which information is displayed to the user and through which the user can provide input to the computer (e.g., interact with user interface elements). Other types of devices can be used to provide interaction with the user. Other devices can be feedback provided to the user, for example, in the form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback). Input from the user can be received, for example, in any form, including sound input, voice input, and / or tactile input.

[0435] The above techniques can be implemented in a distributed computing system including back-end components. The back-end components can be, for example, data servers, middleware components, and / or application servers. The above techniques can be implemented in a distributed computing system including front-end components. The front-end components can be, for example, client computers with graphical user interfaces, web browsers with which users can interact using example implementations, and / or other graphical user interfaces for transmission devices. The components of the system can be interconnected by digital data communications (e.g., communication networks) in any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, wired networks, and / or wireless networks.

[0436] The system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0437] Packet-based networks may include, for example, the Internet, carrier Internet Protocol (IP) networks (e.g., local area networks (LANs), wide area networks (WANs), campus area networks (CANs), metropolitan area networks (MANs), home area networks (HANs), dedicated IP networks, IP private branch exchanges (IPBXs), wireless networks (e.g., radio access networks (RANs), 802.11 networks, 802.16 networks, general packet radio service (GPRS) networks, HiperLANs), and / or other packet-based networks. Circuit-based networks may include, for example, public switched telephone networks (PSTNs), private branch exchanges (PBXs), wireless networks (e.g., RANs, Bluetooth, code division multiple access (CDMA) networks, time division multiple access (TDMA) networks, global system for mobile communications (GSM) networks, and / or other circuit-based networks.

[0438] The computing device may include, for example, a computer, a computer with a browser device, a telephone, an IP phone, a mobile device (e.g., a cellular phone, a personal digital assistant (PDA) device, a laptop computer, an email device), and / or other communication devices. The browser device includes, for example, a browser with a World Wide Web browser (e.g., the Microsoft Windows 10 browser available from Microsoft Corporation). Internet Available from Mozilla Corporation Firefox) is installed on a computer (e.g., a desktop computer, a laptop computer). The mobile computing device includes, for example or other smartphone devices.

[0439] Although many changes and modifications to the present disclosure will undoubtedly become apparent to those of ordinary skill in the art after reading the foregoing description, it should be understood that the specific embodiments shown and described in an illustrative manner are in no way intended to be considered limiting. Further, the subject matter has been described with reference to specific embodiments, but changes within the spirit and scope of the present disclosure will occur to those skilled in the art. Note that the above examples are provided for the purpose of explanation only and are in no way to be construed as limitations of the present disclosure.

[0440] Although the present disclosure has been described with reference to particular embodiments, the disclosure is not intended to be limited to the specific details disclosed herein; on the contrary, the disclosure extends to all variations and generalizations thereof that would be obvious to one of ordinary skill in the art, including those within the broadest scope of the appended claims.

Claims

1. A system comprising a processor and a non-transitory storage medium containing processor-executable instructions, the processor-executable instructions being configured to cause the processor to: Receive cardiovascular medical imaging datasets; A first machine learning algorithm is used to characterize a set of cardiovascular biological properties, wherein: training the first machine learning algorithm based on data including imaging data of a carotid vessel; identifying one or more medical conditions based on the characterization of the set of biological characteristics using a second machine learning algorithm; as well as The set of biometrics, the identification of the one or more medical conditions, or both are output.

2. The system according to claim 1, wherein: The biological characteristic includes (i) plaque in the vessel wall, (ii) stenosis of the vessel wall, (iii) dilation of the vessel wall, or (iv) vessel wall thickness.

3. The system according to claim 1, wherein: The identification of the one or more medical conditions also includes identification of (i) ischemia, (ii) fractional flow reserve (FFR), or (iii) high risk phenotype (HRP).

4. The system according to claim 1, wherein: The first machine learning algorithm is trained on data comprising histology-based annotations of resected tissue.

5. The system according to claim 1, wherein: The first machine learning algorithm is trained based on data including the cardiovascular imaging data.

6. The system according to claim 1, wherein: The second machine learning algorithm is trained based on data comprising a non-radiology or non-imaging data source.

7. The system according to claim 1, wherein: The second machine learning algorithm is trained based on data including physical pressure line readings.

8. The system according to claim 1, wherein: The first machine learning algorithm, the second machine learning algorithm, or both the first machine learning algorithm and the second machine learning algorithm are deep learning algorithms.

9. A method for implementing a hierarchical analysis framework, the method comprising: Receive cardiovascular medical imaging datasets; characterizing a set of cardiovascular biological characteristics using a first machine learning algorithm, wherein the first machine learning algorithm is trained based on data including imaging data of a carotid vessel; identifying one or more medical conditions based on the characterization of the set of biological characteristics using a second machine learning algorithm; and The set of biometrics, the identification of the one or more medical conditions, or both are output.

10. The method according to claim 9, wherein: The biological characteristic includes (i) plaque in the vessel wall, (ii) stenosis of the vessel wall, (iii) dilation of the vessel wall, or (iv) vessel wall thickness.

Images