Anatomical and functional assessment of coronary artery disease using machine learning

Abstract

Anatomical and functional assessment of coronary artery disease (CAD) is performed using machine learning and computational modeling techniques that employ a non-invasive fractional flow reserve (FFR) quantification method based on anatomical and hemodynamic data derived from angiography to perform image segmentation and flow assessment in dependence on a machine learning algorithm, and calculating the FFR in dependence on an accurate computational fluid dynamics (CFD) simulation based on physical properties.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Application No. 17 / 861,100, filed July 8, 2022, entitled “Anatomical and Functional Assessment of CAD Using Machine Learning,” the entire disclosure of which is hereby expressly incorporated herein by reference. Technical Field

[0003] This invention generally relates to the fully automated detection of coronary vessels and their branches in angiography, and more specifically to the calculation of the diameter of such vessels, the detection of stenosis, and the determination of the stenosis percentage and the functional flow restriction caused by the stenosis. Background Technology

[0004] The background description provided herein is for the purpose of presenting the overall context of this disclosure. To the extent described in this background section, the work of the currently attributed inventors, and aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor implicitly acknowledged as prior art to this disclosure.

[0005] In the United States, coronary artery disease (CAD) is one of the leading causes of death, affecting more than 15 million Americans. CAD is characterized by the buildup of plaque in the coronary arteries caused by atherosclerosis, which causes the coronary arteries to narrow (also known as stenosis) or become blocked, and can cause symptoms such as angina and, possibly, myocardial infarction.

[0006] In this paper, 'vascular obstruction state' refers to a condition commonly understood as CAD, such as epicardial coronary artery constriction (localized or diffuse) as seen in imaging studies and characterized by anatomical or functional indicators. Conversely, 'microvascular disease state' refers to disease in the coronary microcirculation characterized by loss of vasodilatory capacity.

[0007] Coronary angiography is the primary invasive diagnostic method for CAD (coronary artery disease), in which contrast agent is injected into the patient's blood vessels through a catheter and imaging is performed to characterize the severity of stenosis. This method relies on visualization of anatomical abnormalities and is semi-quantitative, as visual examination simply estimates the percentage reduction in lumen area. If the estimated diameter reduction is 70% or greater, further evaluation or revascularization procedures, such as coronary stent placement, are usually performed.

[0008] A more physiologically-based and quantitative approach to assessing coronary artery stenosis is the calculation of the fractional flow reserve (FFR), a measure defined as the ratio between the congestion flow in a diseased artery and the expected congestion flow in the same artery without stenosis. In the coronary arteries, for example, FFR can be expressed as the ratio of distal coronary pressure to proximal coronary pressure. For example, an FFR below 0.80 indicates severe stenosis requiring revascularization due to impaired blood flow to the distal vascular bed. Revascularization decisions incorporating FFR have been shown to improve outcomes compared to angiography alone. For example, FFR determination is robust to different patient geometries, taking into account the contributions of collateral vessels to flow and lesion geometry.

[0009] Despite its benefits, FFR is not typically measured by healthcare professionals because of the invasive nature of catheter-based pressure measurements. Some studies have found that in nearly two-thirds of cases, physicians choose not to perform FFR testing, citing patient risks, lack of resources, and additional costs. Another drawback of FFR is its variability due to varying hemodynamic conditions within the patient.

[0010] There is a need for more accurate and less invasive CAD diagnostic techniques. More specifically, there is a need for non-invasive, user-independent FFR measurement methods that do not pose a risk to patients.

[0011] Recently, non-invasive computational workflows for determining FFR and assessing CAD severity have been proposed. These efforts take one of two distinct approaches. The first relies on computed tomography angiography (CTA) data to reconstruct the 3D geometry of the coronary artery tree. Edge detection or machine learning algorithms are used to segment the vessel lumen, and the output is manually corrected by experts. Computational fluid dynamics (CFD) simulations are run on the CTA-derived vessel geometry, and the FFR is calculated on the company's servers or in the cloud. While this non-invasive approach has shown promising results in clinical trials, the use of CTA is severely hampered by imaging artifacts caused by calcification, making it difficult to delineate the boundaries of the vessel lumen. Compared to angiography, CTA's ability to detect small, continuous lesions or capture fine vessel geometry is also limited due to its lower spatial resolution. Finally, CTA data does not provide information about flow, and therefore the boundary conditions (BC) for CFD analysis of hemodynamics often rely on morphometric or population data, and are therefore non-patient-specific.

[0012] The second approach, also based on a non-invasive computational workflow, relies on multiplanar angiography data to reconstruct vascular geometry before performing physics-based flow simulation. The main benefits of using angiography are its superior ability to detect vascular luminal boundaries in the presence of calcified stenosis, and its higher spatial resolution compared to CTA, thus improving the sensitivity of the geometry reconstruction algorithm. Furthermore, time-resolved angiography provides information about how contrast agent moves through the vessel and can therefore be used to estimate blood flow velocity and inform BC for CFD analysis. However, a fundamental challenge of angiography-based FFR quantification methods is reconstructing the 3D geometry of the vessel of interest from a set of 2D images obtained over time and at different positions on the patient's examination table. In addition, all angiography-based FFR quantification methods result in a workflow requiring significant operator input to identify the vessel of interest and assist in segmentation. Finally, all angiography-based methods consider reconstruction of a single coronary artery or model flow physics using highly simplified methods. These drawbacks effectively offset the benefits of using high-resolution angiography data, the most frequently performed procedure in CAD diagnostics.

[0013] Regardless of the method, due to the aforementioned limitations on image data quality, lack of information about flow rates, requirement for operator input, and computational modeling assumptions, all computationally derived FFR methods exhibit poor predictive performance near the critical diagnostic value of FFR=0.8. Therefore, there is currently no pipeline for accurate FFR calculation that can be effectively deployed to any hospital nationwide.

[0014] There is a great need for more accurate and less invasive CAD diagnostic technologies that utilize fully automated, non-invasive FFR measurement methods. Summary of the Invention

[0015] This paper presents a technique for anatomical and functional assessment of coronary artery disease (CAD) using machine learning and computational modeling. By employing a novel method for quantifying fractional flow reserve (FFR) based on non-invasive anatomical and hemodynamic data obtained from angiography, relying on machine learning algorithms for image segmentation and flow assessment, and depending on accurate computational fluid dynamics (CFD) simulations based on physical properties to calculate FFR, this technique addresses the shortcomings of conventional CAD assessment.

[0016] In an example embodiment, the present invention provides a process for assessing the anatomical and functional severity of CAD using static and dynamic coronary angiography data through customized machine learning and computational modeling methods, with the ultimate goal of determining FFR in a less risky and more accurate manner.

[0017] Over the past few years, the use of functional biomarkers for disease has seen significant development, replacing simpler anatomically based markers. In cardiology, fractional flow reserve (FFR), a hemodynamic indicator (i.e., the normalized pressure gradient under maximal flow conditions), has shown better diagnostic results compared to anatomically based markers. In the example embodiments of this paper, two-dimensional (2D) angiographic data is used, specifically dynamic angiographic data that provides a description of the delivery of contrast agent (dye) along the vessel of interest. The 2D time-resolved angiographic data is used to inform computational simulations, resulting in more accurate FFR predictions than in the absence of dynamic contrast agent delivery along the vessel. Furthermore, in the example embodiments, three-dimensional (3D) geometric models are generated from the 2D angiographic data, and those 3D geometric models are used to simulate hemodynamic conditions, including FFR.

[0018] According to an example, a computer-implemented method is provided for generating enhanced segmented images to determine the occlusion status within a vascular examination region. The method includes: receiving, by one or more processors, angiographic image data of a vascular examination region containing a vascular tree, the angiographic image data comprising one or more two-dimensional (2D) angiographic images; applying, by the one or more processors, the angiographic image data to the angiographic image data of a vascular segmentation machine learning model trained to generate segmented images of the vascular tree; applying, by the one or more processors, the angiographic image data of a stenosis machine learning model trained to identify and segment stenosis within the vascular tree to generate segmented patch images; determining, by the one or more processes, the degree of stenosis in each segmented patch image within the vascular tree; stitching each segmented patch image to the segmented images of the vascular tree by the one or more processors to form an enhanced segmented image of the vascular tree; and storing the enhanced segmented image and the determined degree of stenosis for reconstructing a three-dimensional (3D) vascular tree model, performing flow extraction, and determining the occlusion status within the vascular examination region. Attached Figure Description

[0019] This patent or application contains at least one color drawing. A copy of this patent or patent application publication with color drawings will be provided by the United States Patent and Trademark Office upon request and payment of the necessary fees.

[0020] The accompanying drawings described below depict various aspects of the systems and methods disclosed herein. It should be understood that each drawing depicts an embodiment of a specific aspect of the disclosed systems and methods, and each drawing is intended to conform to one or more possible embodiments thereof. Furthermore, wherever possible, the following description refers to the reference numerals included in the following drawings, wherein features depicted in the plurality of drawings are indicated by consistent reference numerals.

[0021] Figure 1 This is a schematic diagram of an example system for performing anatomical and functional assessments of coronary artery disease (CAD) by using a machine learning framework to evaluate angiographic images.

[0022] Figure 2 It is based on an example, such as... Figure 1 A schematic diagram of the CAD evaluation machine learning and computational modeling framework implemented in the system.

[0023] Figure 3 This is a flowchart illustrating the process of assessing coronary artery disease based on case studies.

[0024] Figure 4 This is a flowchart illustrating a method for generating two-dimensional (2D) segmented vascular images from angiography image data, based on an example. This method can be executed by a vascular segmentation machine learning model. Figure 3 It is part of the method.

[0025] Figure 5 Showing Figure 4 Example configuration of a machine learning model for blood vessel segmentation.

[0026] Figure 6A It is based on the instance from Figure 4 The flowchart illustrates a series of example methods for generating a three-dimensional (3D) segmented vascular tree model from a 2D segmented vascular image. Figure 6B It is based on the instance used from, such as, can be by Figure 4 The flowchart shows another example method for generating a three-dimensional (3D) segmented vascular tree model from a 2D segmented vascular image generated during the process. Figure 6C This demonstrates how, based on the instance, it can be... Figure 6A and 6B The process generates example 3D and 1D segmented vascular tree models.

[0027] Figure 7 This paper demonstrates the process of using a graph theory-based reduced-order model that can accurately and efficiently characterize the functional state of CAD based on the definition of instances.

[0028] Figure 8The images show clinical angiography images that, based on examples, can be used to train or predict machine learning models for vessel segmentation and flow extraction.

[0029] Figure 9 The images show synthetic angiography images that can be used to train machine learning models for vessel segmentation and flow extraction, based on examples.

[0030] Figure 10A and 10B It demonstrates example 3D vascular tree models and various lumped parameter models that can be used by computational fluid dynamics system models for each parameter in the parameters.

[0031] Figure 11A-11F Together, they demonstrate an example workflow for anatomical and functional evaluation of generated CAD based on examples.

[0032] In addition to its use in workflows for vascular tree reconstruction, flow data generation, and computational fluid dynamics machine learning models, Figure 12 An example workflow is demonstrated that performs narrow level localization and segmentation based on instances to improve narrow measurement and enhance the generation of 2D segmented images.

[0033] Figure 13 This demonstrates how, based on the instance, it can be... Figure 12 The workflow executes example configurations of angiography-level segmentation neural networks and stenosis-level segmentation neural networks for generating enhanced 2D segmented images.

[0034] Figure 14 Demonstrates based on examples Figure 13 Example operations of the angiography-level segmentation neural network and the stenosis-level segmentation neural network.

[0035] Figure 15 This demonstrates how, based on examples, it can be done. Figure 14 The example operation is an instance of patch smoothing imaging processing that can be performed before or after patch stitching.

[0036] Figure 16 It is based on an example, such as... Figure 12 The flowchart illustrates the process of generating enhanced 2D segmented images to determine the occlusion status within a vascular examination area.

[0037] Figure 17A and 17B Together, they demonstrate example processes for extracting traffic data from the entire vascular tree or a single branch of interest, based on the instance, such as those that can be executed by a traffic data generator.

[0038] Figure 18A and18B This demonstrates the state of being in training mode based on the instance. Figure 18A ) and production mode ( Figure 18B Example architecture of a multi-fidelity neural network.

[0039] Figure 19 This is a flowchart illustrating a method for analyzing vascular health by using a multi-fidelity neural network to determine the state of blockage in a vascular examination area. Detailed Implementation

[0040] This invention provides techniques for anatomically and functionally assessing intravascular obstruction status (e.g., assessing coronary artery disease (CAD)) by analyzing dynamic angiographic image data, using physics-based neural network machine learning, high-fidelity model reduction (e.g., using graph theory and other machine learning techniques), and machine learning-based anatomical segmentation and high-fidelity hemodynamic analysis (e.g., 3D Navier-Stokes), resulting in automated workflows that provide superior diagnostic performance. It also provides techniques for performing non-invasive fractional flow reserve (FFR) quantification based on angiographic data, image segmentation relying on machine learning algorithms, and accurate functional assessment of vessels relying on physics-based machine learning and computational fluid dynamics (CFD) simulations. In some instances, two-dimensional dynamic angiographic data is used to capture dye delivery along the vessel of interest. This dynamic information can be used to inform computational simulations and thus obtain accurate predictions of FFR, particularly in critical cases. Furthermore, although angiographic data does not provide three-dimensional anatomical information, the present invention includes a process of deploying image reconstruction algorithms to obtain 3D and 1D geometric models of the patient's vascular system, which are then used for computer simulations of hemodynamics. Additionally, the present invention includes an improved vascular segmentation process that constructs enhanced vascular segmentation images by performing angiographic image-level vascular segmentation, stenosis localization, and stenosis level segmentation; these vascular tree reconstruction algorithms then use the enhanced vascular segmentation images to generate 3D and 1D geometric models. In such improved processes, the 3D and 1D geometric models contain a more accurate anatomical description of the stenosis. While the techniques described herein relate to the determination of FFR, the same techniques can be used to calculate other flow reserve measures or other hemodynamic values. Therefore, in the examples herein, references to the determination of FFR include the determination of instantaneous waveless ratio (iFR), quantitative fractional flow (QFR), etc.

[0041] In some instances, systems and methods for assessing coronary artery disease are provided. The system can receive angiographic image data of a vascular examination region of a subject, the vascular examination region corresponding to a region of image data containing a vascular tree structure. The angiographic image data may contain multiple angiographic images captured within a sampling time period. The system can apply this angiographic image data to a first machine learning model, namely a vessel segmentation machine learning model. The vessel segmentation machine learning model can generate two-dimensional (2D) segmented vascular images of the vascular examination region and generate from these 2D segmented vascular images a 3D geometric vascular tree model modeling the vessels with the vascular examination region. In other instances, a 1D equivalent vascular tree model can be generated from the 3D vascular tree model. The 3D or 1D geometric vascular tree model can be applied to a second machine learning model, namely a flow extraction machine learning model, to assimilate flow data of one or more vessels within the vascular examination region within the sampling time period. Based on the assimilated flow data and the 3D or 1D geometric vascular tree model, a computational fluid dynamics machine learning model is configured to determine the vascular state in the vascular system, wherein those states may include vascular occlusion and / or microvascular disease / resistance states. Specifically, to determine microvascular disease / resistance, angiographic images can be acquired in two (2) different hemodynamic states: a baseline state and a congested (high-flow) state, and comparisons can be made between the two. In other instances, the microvascular system can be assessed solely by examining angiographic images captured during the congested state. As used herein, vascular occlusion status refers to the determination of vascular health status, including assessments of vascular health based on anatomical features (e.g., determined from angiographic images) and functional features (e.g., determined using physical models and including flow data). Therefore, as used herein, occlusion status should be interpreted as including, but not limited to, degree of stenosis, stenosis length, various hemodynamic parameters (such as FFR), and combinations thereof.

[0042] exist Figure 1In this document, the CAD assessment system 100 includes a computing device 102, or a "signal processor" or "diagnostic device," configured to collect angiographic image data from a patient 120 via an angiography imaging device 124, according to the functions performed in the disclosed embodiments. As shown, the system 100 may be implemented on the computing device 102, and specifically on one or more processing units 104 that may represent a central processing unit (CPU) and / or on one or more graphics processing units (GPUs) (including clusters of CPUs and / or GPUs), either of which may be cloud-based. The features and functions described for the system 100 may be stored on and implemented from one or more non-transitory computer-readable media 106 of the computing device 102. The computer-readable media 106 may include, for example, an operating system 108 and a CAD machine learning (deep learning) framework 110 having elements corresponding to the deep learning framework described herein. More generally, the computer-readable media 106 may store trained deep learning models, including a blood vessel segmentation machine learning model, a flow extraction machine learning model, graph theory or other neural network-based reduced-order models, executable code, etc., for implementing the techniques herein. Computer-readable medium 106 and processing unit 104 may store image data, segmentation models or rules, fluid dynamics classifiers, and other data described herein in one or more databases 112. As discussed in the examples herein, CAD machine learning framework 110 (e.g., various neural networks) applying the techniques and processes described herein can generate 3D and / or 1D segmented vascular tree geometry models, FFR and other fluid dynamic assessments, vascular occlusion status data (such as stenosis degree), and / or microvascular disease data.

[0043] The computing device 102 includes a network interface 114 communicatively coupled to network 116 for communicating with and / or communicating from portable personal computers, smartphones, electronic documents, tablets and / or desktop personal computers or other computing devices. The computing device further includes an I / O interface 115 for connecting to devices such as a digital display 118, a user input device 122, etc. As described herein, the computing device 102 generates indications of the subject's CAD, which may include the state of blood vessels in the vascular system, such as vascular occlusion status (anatomical and functional, calculated via FFR, iFR, or QFR) and microvascular disease prediction status (by comparing changes in distal resistance recorded in two hemodynamic states), as an electronic document that can be accessed and / or shared on network 116.

[0044] In the illustrated example, computing device 102 is communicatively coupled to electronic medical record (EMR) database 126 via network 116. EMR 126 may be a network-accessible database or a dedicated processing system. In some instances, EMR 126 includes data about one or more corresponding patients. EMR data may include vital signs data (e.g., hemoglobin oxygen saturation derived from pulse oximetry, heart rate, blood pressure, respiratory rate), laboratory data such as complete blood count (e.g., mean platelet volume, hematocrit, hemoglobin, mean corpuscular hemoglobin, mean corpuscular hemoglobin concentration, mean corpuscular hemoglobin volume, white blood cell count, platelets, red blood cell count, and red blood cell distribution width), laboratory data such as basal metabolic rate (e.g., blood urea nitrogen, potassium, sodium, glucose, chloride, CO2, calcium, creatinine), demographic data (e.g., age, weight, race and sex, postal code), less common laboratory data (e.g., bilirubin, partial thromboplastin time, international normalized ratio, lactate, magnesium, and phosphorus), and any other suitable patient indicators that are currently available or will be developed in the future (e.g., O2 use, Glasgow Coma Scale (GCS) or components thereof, and urine output in the past 24 hours, antibiotic administration, blood transfusion, fluid administration, etc.); and calculated values, including the shock index and mean arterial pressure. EMR data may additionally or alternatively include chronic medical and / or surgical conditions. EMR data may include historical data collected from previous patient examinations, including historical FFR, iFR, or QFR data. Determination of stenosis, prediction of vascular disease, vascular resistance, CFD simulation data, and other data will be generated according to the techniques described herein. EMR 126 can be updated as new data is collected from angiography imaging device 124 and evaluated using computing device 102. In some instances, the techniques can provide continuous training of EMR 126.

[0045] In conventional angiography imaging applications, angiographic images are captured by a medical imager and then sent to an EMR for storage and further processing, including image processing in some instances, before being sent to medical professionals. In the case of the present invention, occlusion status and microvascular disease status can be determined at the computing device based on angiographic images without first offloading these images to an EMR 126 for processing. In general, the techniques proposed herein significantly reduce the analysis time for cardiologists, partly because they bypass the processing of the EMR 126. The EMR 126 can simply polarize the data during analysis at the computing device 102 and be used to store status determinations and other calculations generated by the techniques described herein. Indeed, the faster and more automated analysis enabled by the present invention offers numerous benefits. For example, vessels corresponding to the left or right coronary tree can be modeled and their occlusion / disease status analyzed individually and sequentially, for example using a 3D or 1D modeler as described herein, while still producing results for cardiologists within minutes.

[0046] In the example shown, system 100 is implemented on a single server. However, the functionality of system 100 can be implemented across distributed devices connected to each other via communication links. In other examples, the functionality of system 100 can be distributed across any number of devices, including portable personal computers, smartphones, electronic documents, tablets, and desktop personal computer devices shown. In other examples, the functionality of system 100 can be cloud-based, for example, one or more connected cloud CPUs or computing systems, labeled 105, customized to perform the machine learning processes and computing techniques described herein. Network 116 can be a public network such as the Internet, a private network such as a private network of a research institution or company, or any combination thereof. The network can include a local area network (LAN), a wide area network (WAN), cellular, satellite, or other network infrastructure, whether wireless or wired. The network can utilize communication protocols, including packet-based and / or datagram-based protocols such as Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), or other types of protocols. In addition, network 116 may include multiple devices that facilitate network communication and / or form the hardware infrastructure of the network, such as switches, routers, gateways, access points (such as the wireless access point shown in the figure), firewalls, base stations, repeaters, backbone devices, etc.

[0047] Computer-readable medium 106 may include executable computer-readable code stored thereon for programming a computer (e.g., including a processor and GPU) to the techniques described herein. Examples of such computer-readable storage media include hard disks, CD-ROMs, digital versatile optical discs (DVDs), optical storage devices, magnetic storage devices, ROMs (read-only memories), PROMs (programmable read-only memories), EPROMs (erasable programmable read-only memories), EEPROMs (electrically erasable programmable read-only memories), and flash memory. More generally, the processing unit of computing device 102 may represent a CPU-type processing unit, a GPU-type processing unit, a field-programmable gate array (FPGA), another type of digital signal processor (DSP), or other hardware logic components that can be driven by a CPU.

[0048] It should be noted that while the example deep learning framework in this paper is described as being configured with an example machine learning architecture, any number of suitable convolutional neural network architectures can be used. More broadly, the deep learning framework in this paper can implement any suitable statistical model (e.g., a neural network or other model implemented via a machine learning process) that will be applied to each of the received images. As discussed herein, this statistical model can be implemented in various ways. In some instances, the machine learning model takes the form of a neural network, support vector machine (SVM), or other machine learning process and is trained using images or cubes to develop a model for vessel segmentation or hydrodynamic calculations. Once these models are sufficiently trained with a series of training images, the statistical model can be applied in real time to analyze subsequent angiographic image data provided as input to the statistical model to determine the presence of CAD and to determine the state of vascular occlusion and disease. In some instances, when implementing the statistical model using a neural network, the neural network can be configured in various ways. In some instances, the neural network can be a deep neural network and / or a convolutional neural network. In some instances, the neural network can be a distributed and scalable neural network. The neural network can be customized in various ways, including providing specific top layers, such as, but not limited to, a logistic regression top layer. A convolutional neural network (CNN) can be considered a neural network containing a set of nodes with bound parameters. A deep CNN can be considered to have a stacked structure with multiple layers. Neural networks or other machine learning processes can include many different sizes, numbers of layers, and connectivity levels. Some layers may correspond to stacked convolutional layers (optionally followed by contrast normalization and max pooling), followed by one or more fully connected layers. The techniques of this invention can be implemented to enable machine learning training using small datasets (e.g., fewer than 10,000 images, fewer than 1,000 images, or fewer than 500 images). In this example, approximately 400 images were used. To avoid overfitting, multiple cross-validation procedures (e.g., 5-fold cross-validation) can be used. In some instances, regularization procedures such as L1 or L2 can be used to avoid overfitting. For neural networks trained on large datasets (e.g., more than 10,000 images), potential overfitting problems can be addressed by using dropout to increase the number of layers and layer size. In some cases, the neural network can be designed to drop fully connected upper layers at the top of the network. By forcing the network to undergo dimensionality reduction in the intermediate layers, it is possible to design neural network models with considerable depth while significantly reducing the number of learning parameters.

[0049] Figure 2An example deep learning framework 200 is shown, which is an instance of the CAD machine learning framework 110 of the computer device 102. The framework 200 includes a 3D / 1D segmented vascular tree geometry model generator 204, which includes a preprocessor level 205 and a vascular segmentation machine learning model 206, which includes two neural networks: an angiography processing neural network (APN) 202 and a second-level semantic neural network 207.

[0050] In the illustrated example, preprocessor 205 receives a clinical angiography image 203A, along with data on contrast agent injections used to form the clinical angiography image. Optionally, preprocessor 205 can be coupled to receive a synthetic angiography image 203B, for example, for machine learning training. Further, preprocessor 205 can be coupled to receive a geometrically adjusted vascular image 203C. In some instances, these inputs can be directly fed into a vessel segmentation machine learning model 206, more specifically into an APN 202. Preprocessor 205 is capable of performing various preprocessing operations on the received image data, which may include denoising, linear filtering, image size normalization, and pixel intensity normalization.

[0051] The deep learning framework 200 can operate in two different modes: a machine learning training mode and an analysis mode. In the framework's machine learning training mode, angiographic image data 203A, synthetic angiographic image data 203B, and / or geometrically adjusted angiographic image data 203C (e.g., horizontal or vertical flipping, arbitrary scaling, rotation, or shearing) can be provided to the APN 202. Depending on the data type and data source, different preprocessing functions and values ​​can be applied to the received image data. In the analysis mode, where the machine learning model has been trained, the captured angiographic image data 203A of the subject is provided to the APN 202 for analysis and CAD determination. In either mode, the preprocessed image data is provided to a 3D / 1D segmented vascular tree geometry model generator 204, which includes a vascular segmentation machine learning model 206 that receives the preprocessed image data and performs the process at the APN 202 and the semantic NN 207, generating 2D segmented vascular images in the analysis model. Therefore, the blood vessel segmentation machine learning model 206 can be a convolutional neural network, such as two different convolutional neural networks with hierarchical configurations, such as... Figure 5 As shown in the examples. Therefore, in some instances, the semantic NN 207 is configured with or without the Deeplab v3+ architecture.

[0052] The 3D / 1D segmented vascular tree geometry generator 204 further includes a 3D modeler 208 configured to generate a 3D vascular tree geometry model of a target region based on a 2D segmented vascular image. As described herein, the 3D modeler 208 is an instance of a 3D reconstruction modeler.

[0053] Once the 3D vascular tree model is generated, generator 204 can apply additional smoothing algorithms and / or surface spline fitting algorithms to further improve the 3D vascular tree model for 3D (e.g., high-fidelity) hemodynamic classification and occlusion analysis.

[0054] To reduce processing time and improve the analysis of vascular occlusion in larger vessels and microvascular disease in smaller vessels, the techniques described here are implemented with a reduced-order model in some instances. In some instances, the 3D segmented vascular tree geometry generated from the captured 2D angiography images is further reduced to generate a 1D segmented vascular tree geometry, while retaining sufficient data for flow data generation, fluid dynamics modeling, FFR, iFR, or QFR determination, and computational fluid dynamics modeling. To implement the model reduction, in some instances, the vascular tree geometry generator 204 includes a 1D modeler 209. The 1D model 209 produces a skeletonization of the 3D segmented vascular tree model, given by path lines / centerlines in 3D space of the vessels included in the 3D segmented vascular tree model and a series of 2D cross-sectional profiles separated at arbitrary distances along each path line / centerline of the tree. An example 1D segmented vascular geometry tree model generated from the 3D segmented vascular geometry tree model is shown below. Figure 6C (1D model 680, which is a vascular tree with M branches, N points, and 4 values ​​for each point) and Figure 7 (1D model 703) is shown. In the example shown, 1D model 680 has M=3 branches and a total of N=21 points, where each point has three Cartesian coordinate values ​​(x, y, z) indicating the point's position in a reference Cartesian coordinate system, and a radius value indicating the radius of the blood vessel determined at that point. For example, the determined radius could be the radius to the inner wall or outer wall of the blood vessel.

[0055] A 3D or 1D vascular tree geometry model from generator 208 or 209 is provided to flow data generator 210, which includes a flow extraction machine learning model 212, which may include at least one of the following types of methods: convolutional neural network (CNN), autoencoder, long short-term memory (LSTM), or graph theory-based flow and pressure reduction model.

[0056] As shown, the flow extraction machine learning model 212 can include many different types of trained and untrained models. In some instances, the flow extraction machine learning model 212 is a deep learning framework proposed by Navier-Stokes, configured to determine pressure and velocity data in 3D or 1D vascular space based on inputs 208 or 209 provided by a modeler. In some instances, the deep learning framework proposed by Navier-Stokes includes one or more methods of the following types: Kalman filtering, physically-informed neural networks, iterative assimilation algorithms based on the time of contrast agent arrival at anatomical landmarks, and TIMI frame counting. Dynamic data describing the delivery of dye along the vessel of interest (e.g., see...) are used. Figure 8 This is used to assimilate information about blood flow velocity. For example, as a dynamic angiographic image showing the progress of fluid flow in the vascular tree of the area being examined, a clinical angiographic image 203A can be provided to a flow data generator 210. In some instances, a contrast agent injection system 203D with precise information about the pressure applied to the dye patch, dye volume, timing, etc., can be used to acquire a dynamic clinical angiographic image of contrast agent movement through the vascular tree, and the image is provided to a flow extraction machine learning model, as also shown in the figure. (See below) Figure 12 As shown, in various instances, the flow data generator only receives dynamic angiography images and generates flow data, which is later combined with data derived from the 3D vascular tree model / 1D vascular tree model.

[0057] More generally, the flow extraction machine learning model 212 is configured to determine assimilated flow data of one or more vessels within a 3D or 1D vascular tree geometry model over a sampling time period. This determination may include determining the pressure and / or flow velocity of multiple connected vessels in the 3D or 1D vascular tree geometry model.

[0058] In some instances, the flow extraction machine learning model 212 determines lumped parameter boundary condition parameters for one or more vessels in a vessel examination region. In some instances, the flow extraction machine learning model 212 determines a lumped parameter model for the flow of a first vessel and a lumped parameter model for the flow of each vessel branching from the first vessel. Any of these can then be stored as assimilated flow data.

[0059] Assimilated flow data from flow data generator 210 and a 3D vascular tree model (or a 1D vascular tree model) are provided to a computational fluid dynamics machine learning model 214, which can apply a physics-based process to determine the vascular occlusion status and / or the microvascular disease status of one or more vessels within the 3D vascular tree model (or 1D vascular tree model) (uniformly designated 216). Although described as a machine learning model in various instances, in other instances, the computational fluid dynamics model is implemented without using machine learning techniques. In some instances, computational fluid dynamics machine learning models include one or more of the following: multi-scale 3D Navier-Stokes simulations using a reduced-order (lumped-parameter) model; reduced-order Navier-Stokes (1D) simulations using a reduced-order model derived from a graph theory framework dependent on a 1D nonlinear theoretical model; reduced-order model simulations (lumped-parameter, 0D) models; or models based on neural networks for a whole segmented vascular tree model, obtained through differences between fundamental fact data (computer-simulated or clinical), including geometry, pressure, flow rate, and indices such as FFR, iFR, or QFR. In the examples shown, according to the examples in this paper, the computational fluid dynamics machine learning model includes at least a 3D high-fidelity model 211 and a reduced-order model 213, which can be a graph theory model, a multi-fidelity neural network model, or other models.

[0060] Compared to occlusion analysis techniques based on high-fidelity 3D technology, any of the techniques used in this paper to define the reduced-order model can produce faster results. Furthermore, the techniques presented here can model and analyze not only large blood vessels but also microvascular systems, thus enabling the determination of occlusion states in large vessels and microvascular disease states in small vessels.

[0061] In some instances, the computational fluid dynamics machine learning model 214 is configured to determine the FFR, iFR, and / or QFR of one or more vessels in a 3D or 1D vascular tree model based on flow data. In some instances, the computational fluid dynamics machine learning model 214 is configured to determine the vascular occlusion status based on the FFR, iFR, and / or QFR of one or more vessels. In some instances, the computational fluid dynamics machine learning model 214 is configured to determine the coronary flow reserve (CFR) of one or more vessels based on flow data, based on one or more physiological states (baseline and congestion), and to determine the microvascular disease status based on the CFR of one or more vessels. Determining the vascular occlusion status includes determining the presence of stenosis in one or more vessels. Determining the microvascular disease status includes determining a lumped parameter model at the boundaries of vessels within a vascular examination area.

[0062] exist Figure 3 The image illustrates a process 300 for assessing coronary artery disease, which can be performed by system 100. At process 302, 2D angiographic image data is acquired by medical imager 116 and provided to computer device 102, during which preprocessing operations may optionally be performed. In training mode, the 2D angiographic image data may be captured clinical angiographic image data, but such training data may further include synthetic image data, geometrically adjusted image data, etc. Figure 2 As shown.

[0063] In the examples, 462 clinical angiography images enhanced by a combination of geometric transformations (scaling, horizontal flipping, vertical flipping, rotation, and / or shearing) are shown (e.g., see...). Figure 8 The machine learning model 206 for vessel segmentation was trained to generate over 500,000 angiographic images and 461 synthetic angiographic images (see, for example). Figure 9 This is enhanced through those geometric transformations into an additional set of over 500,000 images. As shown, clinical angiography images can include time-series data across multiple frames, which are segmented to extract velocities throughout the vascular tree. In some instances, the vessel segmentation machine learning model 206 includes a synthetic image generator configured to generate synthetic images using combinations of transformations such as flipping, shearing, rotation, and / or scaling. In any case, these numbers of images are provided only through empirical examples, as any suitable number of captured or synthesized training images can be used. In training mode, image data is fed into the CAD evaluation machine learning framework 110 to generate two types of models: the vessel segmentation machine learning model 206 and the flow extraction machine learning model 212. In diagnostic mode, image data is fed into the CAD evaluation machine learning framework 110 for classifying and diagnosing CAD (in diagnostic mode, image data is also fed into machine learning model 206 and flow extraction machine learning model 212).

[0064] At process 304, the CAD evaluation machine learning framework 110 applies the received image data to the blood vessel segmentation machine learning model, for example, through the APN 202 and semantic NN 207 of the blood vessel segmentation machine learning model 206, and generates a 2D segmented blood vessel image. At process 306, the CAD evaluation machine learning framework 110 receives the 2D segmented blood vessel image, for example, through a 3D modeler 208, and generates a 3D segmented blood vessel tree model or a 1D segmented blood vessel tree model.

[0065] At process 308, a 3D segmented vascular tree model or a 1D segmented vascular tree model is applied to the flow extraction machine learning model 212 of the flow data generator 210, and assimilated flow data is generated for one or more vessels in the 3D vascular tree model or the 1D segmented vascular tree model during the sampling period.

[0066] At process 310, assimilated flow data and a 3D or 1D segmented vascular tree model are applied to a computational fluid dynamics machine learning model 214. This model uses the 3D or 1D segmented vascular tree model to evaluate the data and determines vascular health by solving 3D Navier-Stokes equations, graph-based reduced-order models, or other neural network reduced-order models, such as determining vascular occlusion status using indices like FFR, iFR, QFR, or other metrics. If data on two hemodynamic states are available (e.g., baseline and congestion conditions), the microvascular disease state or CFR is determined based on the lumped parameter values ​​of the boundary conditions for each hemodynamic state.

[0067] Figure 4 The process 400 shown is an example implementation of process 304, which generates a 2D segmented vascular image from angiographic image data. This process can be performed by an angiography processing network 202 and a semantic NN 207 of a vascular segmentation machine learning model 206. Initially, at process 302, 2D angiography image data of the target vascular region is received. At process 402, image preprocessing (including image size normalization and / or pixel intensity normalization) is applied before further image processing by an angiography processing network implemented with a convolutional neural network (e.g., APN 202). This further preprocessing may include nonlinear filtering, denoising, and / or contrast enhancement, which, when combined with process 404, filter out objects such as ducts and bone structures. That is, at process 404, the preprocessed 2D angiography image is applied to a second convolutional neural network (CNN) (e.g., semantic NN 207) trained using clinical angiography image data, synthetic image data, and / or geometrically altered image data. Figure 5An example CNN framework 500 is shown, configured with a combination of an angiography processing network (APN) 503 and a semantic neural network in the form of a Deeplab v3+ network 507, configured to generate 2D segmented vessel images at process 408. The Deeplab v3+ network 507 is a semantic segmentation deep learning technique comprising an encoder 502 and a decoder 504. A series of angiography images 501 are fed as input to the angiography processing network 503, which consists of several 3×3 and 5×5 convolutional layers, applying nonlinear filters to perform contrast enhancement, boundary sharpening, and other image processing functions. The APN 503, together with the semantic neural network 507, forms a vessel segmentation machine learning model, as shown in model 206. The APN 503 feeds the encoder 502, which applies dilated convolutions to the image data and employs a rate to control the effective field of view of each convolutional layer. A higher rate allows for a larger image capture area for convolutions and low-level feature extraction that can be performed on each input image. For example, rates of 6, 12, and 18 can be used to influence different fields of view to capture different features, such as features with different resolutions. Using dilated or attenuated convolutions, dilated samples of an image or a portion of an image are convolved into a smaller image. Encoder 502 uses several attenuated convolution strides to determine low-level features, then applies 1×1 (depth-separable) convolutions to combine the outputs of many different attenuated convolutions. This produces a single matrix of features belonging to different categories, which, along with high-level features determined by the 1x1 convolutions applied to the original input image, is input to decoder 504. In some instances, attenuated or depthwise convolutions can be used to perform the convolutions of encoder 502. Decoder 504 concatenates the low-level and high-level features into a stacked matrix, then applies transposed convolutions to this matrix, thereby assigning a category label to each pixel using a fractional stride size based on spatial and feature information. These transposed convolutions generate a probability map that determines the likelihood of each pixel belonging to the background or blood vessel category. As shown, a Softmax function is applied to generate the output segmented 2D image 505.

[0068] exist Figure 6A The diagram illustrates a process 600 that can be executed by a 3D segmented vascular tree model generator 204 to generate a 3D segmented vascular tree model from a set of generated 2D segmented vascular images 602. As shown, according to an example, any one of four different pipelines can be used to generate the 3D segmented vascular tree model, from which a 1D segmented vascular tree model can also be obtained.

[0069] In the first configuration, at process 603, the 3D modeler 208 receives 2D segmented vessel images and locates the centerline of each 2D segmented vessel image. At process 604, the 3D modeler 208 uses geometric tools to co-localize points from each 2D segmented vessel image, said geometric tools may include epipolar geometry, projective geometry, Euclidean geometry, or any relevant geometric tools, and at process 606, triangulation is performed on the 3D points having projections (backprojections) mapped onto the co-localized points. The local radius of the vessel is determined based on each 2D segmented vessel, and these radius vectors are projected onto the 3D centerline. From there, at process 608, the 3D modeler 208 determines the vessel contour based on the triangulated 3D points, and generates a 3D vessel tree model at process 610. From there, a 1D vessel tree model is generated at process 620.

[0070] In the second configuration, at process 612, the 3D modeler 208 generates multiple 3D rotation matrices from the 2D segmented vascular images. Then, at process 614, the 3D modeler 208 generates a 3D segmented vascular tree model by solving a system of linear least squares equations that map the multiple 2D segmented vascular images to 3D space.

[0071] In the third configuration, at process 616, the 3D modeler 208 projects voxels in 3D space forward onto multiple 2D segmented blood vessel images, and at process 618, identifies the 3D voxel sets projected onto the multiple 2D segmented blood vessel images. The resulting binary volume is then smoothed to ensure realistic blood vessel boundaries.

[0072] In the fourth configuration, an active contour model is used to reconstruct the 3D geometry of the blood vessels. At process 622, the final centerline of each 2D segmented blood vessel image is established. At process 624, the endpoints of each blood vessel are identified in the multiple 2D segmented blood vessel images and back-projected onto them to identify endpoints in 3D space. At process 626, a 3D cylinder is drawn between these 3D endpoints, and external and internal forces on the cylinder are defined as contour modeling based on material properties, system imaging parameters, and reprojection errors between the cylinder and the multiple 2D segmented images for deformation, until the projection matches the blood vessel shape in each 2D image. At process 628, reprojection deformation is performed, forcing the 2D image into a 3D cylinder that can be deformed. The forces deform the cylinder to minimize the reprojection error. Process 628 can be repeated for all branches of the blood vessel until the complete coronary artery tree is reconstructed.

[0073] exist Figure 6BIn the fifth configuration shown (process 600), a Euclidean distance transform (process 650) is applied to the segmented vessel image 602 to encode the 2D diameter of each branch. At process 652, the set of segmented vessel images with the Euclidean transform from process 650 is fed into a 3D vessel tree reconstruction machine learning model trained to reconstruct a 3D vessel tree. The representation of such a tree can be either 3D (610) or 1D (620) generated directly from the machine learning model of process 652.

[0074] Figure 6C Examples of the 3D vascular tree model 682 and the 1D vascular tree model 680 discussed above are shown. In the examples shown, the vascular tree i is composed of the tree matrix M. i ×N i ×4 represents, where M i N is the number of branches in vascular tree i. i This represents the number of points on the centerline of each branch, and 4 represents the data encoded at each point on the branch centerline, specifically its three-dimensional coordinates (x, y, z) and radius r. The 3D vascular tree model 682 consists of M covering all branches. i The smoothed analysis surface of the occupied volume is given. The 1D vascular tree model 680 consists of M branches. i Each point N i The centerline coordinates and radius values ​​are given.

[0075] In some instances, the flow extraction machine learning model 212 of this paper is implemented as a physical information neural network, specifically a neural network capable of encoding any fundamental physical laws governing a given dataset and which can be described by partial differential equations. For example, the flow extraction machine learning model may include partial differential equations for 3D Navier-Stokes equations, a set of four partial differential equations for mass and momentum balance, and an unknown field that is a 3D velocity vector (v x v y v z ) and scalar p.

[0076] In the example, the solution for the flow within the 3D vascular tree model is generated using the incompressible 3D Navier-Stokes equations of flow, based on equations (1) and (2):

[0077]

[0078] (1) and (2)

[0079] Where u is the fluid velocity, p is the pressure, and f is the body force (assumed to be zero here), and It is the viscous stress tensor of a Newtonian fluid.

[0080] Alternatively, in some instances, the traffic extraction machine learning model 212 may include a graph-based reduced-order model, such as... Figure 7 The graph-based reduction model can be run in either machine learning or analytical mode. In machine learning mode, a dense graph 702 is defined using training data 701 of measurements on flow rate, narrow geometry, FFR, iFR, QFR, etc., to represent the difference between the fundamental facts data given by the 1D nonlinear theoretical model 703 and the low-fidelity flow model. The fundamental facts data can be provided by computer simulation of the 3D Navier-Stokes equations 704 or by in vivo data obtained in a catheter insertion laboratory 705. The dense graph 702 of the difference can be generated by analyzing numerous permutations of narrow diameter, length, eccentricity, flow rate through the narrow, or combinations thereof. Once the dense graph of the difference is generated, the reduction model 706 can be derived by nonlocal calculus, deep neural networks, or direct traversal of the vertices of the graph. The reduction model can be an algebraic equation, or an ordinary differential equation or a partial differential equation. In analytical mode, the input 707 to the reduced-order model is the geometry given by the 1D segmented vascular tree extracted by process 610, and the boundary conditions regarding flow rate and pressure extracted by the data assimilation process 308. The reduced-order model 706 and the input 707 produce the desired vascular occlusion state 708 (anatomical and functional, calculated via FFR, iFR, or QFR) and microvascular disease state. The 1D nonlinear theoretical model 703 is a 1D nonlinear Navier-Stokes model, which includes two partial differential equations for mass balance and momentum balance, with the unknowns being flow rate Q and cross-sectional mean pressure P.

[0081] In the example, the solution for the flow rate obtained using the 1D nonlinear theoretical model is generated by using the conservation of mass and momentum of incompressible Newtonian fluids according to the following set of equations, namely equations (3) and (4).

[0082] (3) and (4)

[0083] Where x is the axial coordinate along the blood vessel, t is time, A(x,t) is the cross-sectional area of ​​the lumen, U(x,t) is the average axial blood flow velocity on the cross-section, P(x,t) is the average blood pressure on the cross-section, and ρ f It is assumed that the blood density is constant, and f(x,t) is the frictional force per unit length. It can be assumed that the momentum correction factor in the convective acceleration term of equation (1) is equal to one. Equations (3) and (4) can also be derived by integrating the incompressible Navier-Stokes equations over the general cross section of the cylindrical domain.

[0084] In any case, the flow extraction machine learning model presented in this paper can be formed by data-driven algorithms to infer solutions to these general nonlinear partial differential equations by substituting physical information for classification models. Information about physical laws governing the time-dependent dynamics of the system, or empirically validated rules or other domain-specific expertise, can be used as a regularizing surrogate to constrain the space of permissible solutions to a manageable size. In turn, encoding such structured information into the machine learning model amplifies the informative content of the data seen by the algorithm, enabling it to quickly steer towards the correct solution and perform well inductively even with only a small number of training samples available. Furthermore, the proposed reduced-order model will be trained using the discrepancy between a low-fidelity 1D nonlinear model of graph theory and blood flow and fundamental factual data given by a 3D high-resolution Navier-Stokes model or in vivo anatomy and hemodynamic data to accurately and efficiently capture the hemodynamics around stenosis. In various instances, reduced-order models are defined from the difference graph using one of three methods: a) CNN, b) nonlocal calculus, and c) exploring the graph using lateral algorithms (see, for example, Banerjee et al., A graph theoretic framework for representation, exploration and analysis on computed states of physical systems, Computer Methods in Applied Mechanics and Engineering, 2019, which is hereby incorporated by reference).

[0085] In this example, the flow extraction machine learning model is configured to include Hidden Fluid Mechanics (HFM), a deep learning framework capable of encoding physical information about a class of physical laws governing fluid motion (i.e., the Navier-Stokes equations), as described in Raissi et al., “Hidden Fluid Mechanics, A Navier-Stokes Informed DeepLearning Framework for Assimilating Flow Visualization Data,” published on the cover of August 13, 2018, which is incorporated herein by reference. In this example, the flow extraction machine learning model applies fundamental conservation laws (i.e., the laws of conservation of mass, momentum, and energy) to infer hidden quantities of interest, such as velocity and pressure fields, solely from a 3D vascular tree model generated from angiographic image data acquired at different times. The flow extraction machine learning model can apply algorithms that are agnostic to geometry or initial and boundary conditions. This makes the HFM configuration highly flexible in selecting the types of vascular image data that can be used for training and diagnosis of the model. Flow extraction machine learning models are trained to predict pressure and velocity values ​​in two-dimensional and three-dimensional flow in imaged blood vessels. This information can be used to determine other physically relevant properties, such as pressure or wall shear stress within the artery.

[0086] In some instances, computational fluid dynamics models are configured to determine the lumped parameter model for each vessel attached to a 3D or 1D vascular tree model. For example... Figure 10A As shown, computational fluid dynamics models can include a series of lumped parameter models (LPMs) for different blood vessels. An example LPM1000 is provided for a cardiac model coupled to the inflow surface of a 3D vascular tree 1002. The vascular tree 1002 is formed by multiple different blood vessels labeled with different letters: A inlet, BH aortic outlet, and ak coronary outlet. LPM 1006 is used for each outlet BH, representing the microcirculation of vessels other than the coronary arteries. LPM 1008 is provided for the coronary outlet ak and coupled to LPM 1000 representing the heart. The parameters of this model are estimated based on the patient's flow and pressure data (measured in a catheter insertion laboratory, or estimated using data assimilation techniques described in 308 or using morphometry factors such as Murray's law). LPM 1004, including both the left and right sides of the heart, can be used to run analyses in a closed-loop configuration. Models of this nature can be used to compute hemodynamics under pulsatile conditions. Figure 10BThe second example LPM is shown. Here, the inflow boundary conditions can be defined by the patient's mean arterial pressure or by the mean flow rate measured or estimated based on dynamic angiography data. Patient-specific outflow boundary conditions can be defined by the flow rate of each vessel estimated by a fluid dynamics machine learning process 308, which assimilates the flow rate data within the sampling period, or by the LPM (resistor) coupled to each outlet face of the coronary tree. Once the computational solutions for pressure and flow rate within the vascular tree are known, the LPM for each branch of the vascular tree (whether 3D or 1D) can also be estimated, assuming the pressure gradient on the LPM decreases to a fixed capillary pressure level. Models of this nature can be used to simulate hemodynamics under steady-state conditions. This process can be repeated for each available hemodynamic state (e.g., baseline and congestion). LPMs are used to represent cardiovascular regions where full details of the flow rate solution are not required, but it is important for the model to include the relationship between pressure, flow rate (and in some cases volume) in these regions. These are ideal regions when detailed spatial information about vascular geometry is neither available nor important. Their parameters have a fundamental impact on the pressure and velocity fields in the vascular tree model, and therefore must be tuned to achieve physiological values ​​consistent with data assimilated from patient images or any other clinical records available for the individual in question. When there is insufficient data available for parameterization of a particular patient, data on expected values ​​from the scientific literature can be used to help determine appropriate parameters.

[0087] In the example, the anatomical and functional evaluation of CAD follows Figure 11A-11F The described workflow is 1100. Figure 11A-11F Together, a workflow of 1100 is formed, in which Figure 11A , 11B 11C is a subplot above workflow 1100, with each subplot connected to the others from left to right, and in which... Figure 11D , 11E 11F and 11F are the lower subplots across workflow 1100, each subplot connecting to the other from right to left, and each subplot connecting to... Figure 11C , 11B And the corresponding sub-images in 11A. Several angiographic images 1101 taken at different orientations are fed into a machine learning segmentation module 1102 (e.g., corresponding to processes 302, 304, 402, and 404), which automatically generates 2D segmented images of the angiographic images 1103. These images are then fed into an algorithm 1104, which generates 3D and 1D segmented vascular tree models 1105, for example, following... Figure 6A and 6BThe process described herein is a combination of procedures. This workflow is used to automatically characterize the luminal diameter within the vascular examination area of ​​angiographic image data 1101 and thus the anatomical severity of CAD. Further, a series of angiographic images 1106 defining dye delivery along the region of interest in the vascular tree are fed into a series of fluid dynamics machine learning algorithms 1107, which assimilate information about blood flow velocity and / or pressure for each vessel of interest in the vascular tree. The system generates boundary conditions 1108 regarding velocity, which, together with the vascular tree 1105, are fed as input for a graph-theory-derived reduced-order model 1109, which ultimately produces desired functional measures of CAD 1110, including FFR, iFR, QFR, and microvascular resistance.

[0088] Workflow 1100 includes a machine learning segmentation module 1102, which provides static angiography image data 1101 and generates a complete 2D segmented image 1103 of the vascular tree in the vascular examination region captured in the image data 1101. This 2D segmented image 1103 then proceeds to the 3D reconstruction process 1104.

[0089] In another instance, the procedures and workflows described herein can be implemented to analyze angiographic image data at the angiography level (i.e., the entire image level) as described above and at the stenosis levels as described below, to generate enhanced 2D vessel segmentation images that can be used to generate 3D and 1D geometric models. In these additional instances, 2D segmentation is enhanced by identifying stenosis, thereby allowing for enhanced segmentation of image data on local regions corresponding to stenosis locations (also referred to herein as “patches”), and allowing for the determination of the degree of stenosis, stenosis length, etc., at each stenosis location within the vessel examination area. Analyzing patch images (also referred to herein as “patches”) allows for more accurate analysis of severe stenosis within the vessel examination area.

[0090] Therefore, existing techniques for enhanced 2D segmentation include machine learning pipelines capable of characterizing stenosis in angiographic image data. These pipelines combine neural networks and image processing algorithms to automatically locate, segment, and measure stenosis locations within an area of ​​vascular examination, and generate enhanced 2D segmented images based on those stenosis locations (e.g., patches). In an example, we compare the sample pipeline to a base-fact measurement and find that the sample pipeline is able to measure a stenosis diameter of 0.206 ± 0.155 mm, or approximately one pixel, as determined by the base-fact measurement.

[0091] Figure 12An example workflow 1200 for generating enhanced 2D segmented image data to determine the occlusion status within a vascular examination area is illustrated. This workflow can be implemented by the systems, processes, and workflows described herein, such as by the aforementioned CAD evaluation system 100 and / or deep learning framework 200. Specifically, workflow 1200 can be used in place of workflow 1100 to provide, among other things, stenosis level analysis of angiographic image data, thereby allowing the generation of a 2D segmented image similar to image 1103 but enhanced based on local stenosis analysis and an estimation of the occlusion status through a reduced-order model derived from a multi-fidelity neural network model.

[0092] In the illustrated example, angiographic image data 1202 containing one or more images of the vascular examination area is fed into a neural network 2D segmentation module 1204 containing two machine learning pipelines (angiographic image-level segmentation neural network 1206 and a stenosis-level segmentation neural network 1208). These two machine learning pipelines are combined to produce an enhanced 2D segmented image 1210, which is then fed into a vascular tree reconstruction module 1212. In the illustrated example, the angiographic image data 1202 may contain real or synthetic X-ray angiographic images.

[0093] The angiography-level segmentation NN 1206 is a neural network trained to analyze the entire angiography image and generate a 2D segmentation image of the vascular tree within the imaged vascular examination region. In this example, the angiography-level segmentation NN 1206 can be implemented by the machine learning segmentation module 1102 (e.g., corresponding to the segmentation performed by the vascular segmentation machine learning model). Figure 3 and 4 The process is as described above. Therefore, the angiography-level segmentation NN 1206 generates 2D segmented images. The angiography-level segmentation NN 1206 can be implemented as an angiography processing network (APN), and then trained to remove ducts, overlapping bone structures, and / or other image artifacts from the corresponding input data images (see, for example, see...). Figure 4 Boxes 402 and 404 and Figure 5 (as discussed in other examples in this article).

[0094] In contrast, the stenosis level segmentation neural network 1208 is trained to receive angiographic image data 1202 and perform stenosis localization and stenosis segmentation to generate segmented patch images corresponding to the stenosis.

[0095] Figure 13An example implementation of the NN 2D segmentation module 1204 is shown. In the example shown, the stenosis level segmentation NN 1208 has a multi-level machine learning architecture. The first NN level 1250 is trained to identify stenosis locations in blood vessels within a vascular examination region of image data 1202. The second NN level 1252 is trained to perform segmentation on these identified stenosis locations, for example, to binarize the image data of the stenosis locations. Similar to 1206, 1102, 503 and 507, 402 and 404, NN level 1252 comprises two levels: an APN followed by a semantic segmentation NN.

[0096] In the example shown, the NN-level 1250 is an object detector, such as the YOLOv5 neural network. Compared to other state-of-the-art object detection networks, the YOLOv5 architecture offers near real-time inference speed and better performance when detecting small objects. However, it will be understood that any number of neural network architectures can be used.

[0097] The NN-level 1252 can be trained to perform segmentation on specific regions corresponding to the narrowness identified at the NN-level 1250. For example, the NN-level 1252 can be a patch-level trained NN that performs segmentation on patch images centered on narrow locations. The NN-level 1252 can contain an APN, followed by a semantic segmentation NN such as VGG UNet or Deeplabv3.

[0098] The narrow-level segmentation NN 1208 further includes a patch insertion stage 1254, a narrow quantization module stage 1256, and a patch smoothing stage 1258. That is, in some instances, the narrow-level segmentation NN 1208 is a module containing NN stages and non-machine learning stages. However, in some instances, the narrow-level segmentation NN can be trained to utilize layers to perform the processes of stages 1254-1258 described in this paper.

[0099] Figure 14An example operation of the NN 2D segmentation module 1204 is shown, which includes an angiography-level segmentation NN 1206 and a stenosis-level segmentation NN 1208 (comprising the following levels: stenosis localization NN 1250, stenosis segmentation NN 1252, patch insertion 1254, stenosis quantization 1256, and patch smoothing 1258). The stenosis localization NN level 1250 has identified three stenosis regions 1302, 1304, and 1306 in the received angiography image data 1300. Based on the training of the machine learning model, the stenosis region can be an actual stenosis region or a candidate stenosis region, which can include both actual and potential stenosis regions. The latter allows for the use of downstream processes to subsequently determine which candidate stenosis regions are actual stenosis regions. In the example shown, the NN level 1250 identifies the stenosis location across the entire image data, and thus the stenosis location across the entire vascular examination area. In the example shown, the NN level 1250 identifies the location of each stenosis by its centroid (e.g., the X and Y coordinates of the angiography image data 1300). After this identification, a patch image centered on the centroid can be generated through NN-level 1250 or through subsequent processes not shown. For example, a 40×40 pixel patch image centered on each centroid (X, Y) can be generated, resulting in patch images 1302a, 1304a, and 1306a corresponding to the narrow positions 1302, 1304, and 1306, respectively.

[0100] Each of the patch images 1302a, 1304a, and 1306a is fed into the narrowing segmentation NN stage 1252 to generate segmented patch images 1302b, 1304b, and 1306b, respectively. That is, the narrowing segmentation NN stage can be trained to segment patch images formed by a reduced set of pixels from image 1300 and centered at the centroid (X, Y), thereby defining the narrowing location. These generated segmented patch images are then provided to the patch insertion stage 1254.

[0101] Patch insertion level 1254 can perform the stitching process by acquiring the segmented patch image and stitching it into the 2D segmented image generated by the angiography film level segmentation NN 1206.

[0102] The stitching of level 1254 can be performed based on the centroid coordinates of the stenosis location defined according to level 1250. In some instances, the patch smoothing level (1258) performs image alignment between the vascular structures (edges, etc.) in the segmented patch image and the regions adjacent to the vascular structures in the 2D segmented image from the angiography film-level segmentation NN 1206. In yet another instance, patch smoothing level 1258 can apply image processing to smooth the segmented image patches and their transitions between adjacent regions. For example, patch smoothing level 1258 can perform a smoothing process based on linear interpolation and a Savitzky-Golay filter, which includes filling, magnification, Gaussian filtering, erosion, reduction, thresholding, and / or boundary smoothing. The end result is an enhanced 2D segmented image 1210. Figure 15 Example smoothing operations applied to a 40×40 pixel patch image and their example sorting are shown, the patch image having been padded to a 50×50 pixel patch image for smoothing. While padding is optional, it allows for better control over the smoothing operation performed on the inserted patch.

[0103] Return to Figure 14The stenosis quantization module 1256 can determine the degree of stenosis, stenosis length, or other quantities, such as stenosis eccentricity, for each of the identified stenosis regions 1302, 1304, and 1306. In the illustration, the stenosis level segmentation NN 1208 is configured with module 1256 receiving patch images from patch insertion level 1254. In other instances, module 1256 may receive patch images directly from NN level 1252 or from patch smoothing level 1258 after smoothing has been performed. In some instances, module 1256 may be configured to determine the stenosis diameter and / or diameter reduction percentage for each stenosis location (e.g., patch image). The degree of stenosis at the stenosis location is provided by comparing the segmented diameter at the stenosis location with the diameter of the vessel portion proximal or distal to the stenosis location. The diameter and these distinct locations can be determined by counting pixels from vessel edge to vessel edge in the segmented image at the corresponding locations. In addition to determining the degree of stenosis, the stenosis quantization module 1256 can also determine the stenosis length for each stenosis location. For example, module 1256 can determine (or continuously) the diameter at various points along the axial direction of the blood vessel and compare these diameters to identify diameters within a certain percentage range, standard deviation, or other statistical measure as corresponding to either a narrow or non-narrow portion of the blood vessel. The quantified data (e.g., degree of stenosis and stenosis length) can then be stored as metadata in the enhanced 2D segmented image 1210 for use in generating a 3D / 1D geometry tree model. It will be understood that any number of image processing techniques (including those other than pixel-count-based techniques) can be used to determine the degree of stenosis, stenosis length, or other measures such as eccentricity. As described above, the stenosis quantification module 1256 can be performed on the patch image before the patch image is stitched onto the 2D segmented image. Stenosis quantification may occur after stitching. Stenosis quantification may occur after patch stitching and smoothing operations.

[0104] In an example implementation, the stenosis quantification module 1256 can use image processing algorithms to measure the stenosis diameter and quantify the percentage reduction in diameter. For example, an Euclidean distance transformation can be performed to create a distance map. A skeletonization algorithm can then be applied to identify the vessel centerline in the patch image. From there, the radius along the vessel segment can be determined by sampling the Euclidean distance map at the location of the centerline pixels. The pixel diameter can then be converted to millimeters using a calibration factor derived from the width of a reference structure (such as a 6F catheter) or metadata of the angiographic image 1300. A signal processing algorithm can then be applied to the list of radii to identify peaks (diameters proximal and distal to the stenosis) and troughs (damage diameter), thereby calculating the stenosis percentage as:

[0105] Stenosis % = 1 - (damage diameter) / (0.5 * (proximal diameter + distal diameter)).

[0106] The reference diameter is the average of the diameters of the proximal and distal sides of the narrow passage.

[0107] Return to Figure 12 In some instances, the input angiography image data may be fed to an image normalization process (not shown) before being provided to the neural network 2D segmentation module 1204. Alternatively, module 1204 may be configured to perform a normalization process. The normalization process can normalize image characteristics such as intensity and contrast. For example, z-score normalization can be used to normalize the angiography image data to increase the robustness of the neural network to images from different datasets, where the distribution of pixel intensity may not match the distribution of the training dataset. In an example implementation of the stenosis localization neural network 1250, the training dataset consists of 974 angiography images with annotations corresponding to each stenosis. Cardiologists manually generate annotations by selecting the centroid of each stenosis in each angiography image. The center of the bounding box covering the stenosis is then defined using centroid coordinates with fixed width and height parameters. Each bounding box annotation is defined by its normalized coordinates (centroid X, centroid Y, width, height). In some instances, the bounding box size is fixed at 40 × 40 pixels. In other instances, the bounding box size can be fixed at 64×64 pixels, 32×32 pixels, or any other combination of sizes. 779 angiography images were used to train the network, and 195 angiography images were used for validation, corresponding to an 80 / 20 training-validation segmentation.

[0108] In this example implementation, transfer learning is employed. The YOLOv5 network architecture used for the NN-1250 is initialized with pre-trained weights from the ImageNet database (https: / / www.image-net.org / ) and trained for 300 epochs using the ADAM optimizer with a learning rate of 1e-3. The loss function is a combination of binary cross-entropy loss and focus loss, which measures the network's ability to predict bounding boxes at the correct locations and classify them as stenosis. The output of the stenosis localization NN-1250 is the normalized coordinates of the bounding boxes (centroid X, centroid Y, width, height), which identify the location of each stenosis from the input angiography image. The input angiography image is then cropped into patches centered on each identified stenosis using the normalized coordinates of the bounding boxes. The patches are then used for further analysis; for example, the patches can be fed into the stenosis segmentation neural network-1252.

[0109] In an example implementation of the stenosis segmentation neural network 1252, the training dataset consists of 965 patches cropped from angiographic images. Each patch is cropped based on the centroid coordinates of the identified stenosis. In some instances, the stenosis can be manually identified by a cardiologist. In other instances, image processing or a neural network can be used to identify the stenosis; one example of a neural network is the stenosis localization neural network 1250. Each patch is accompanied by a corresponding annotation. Annotations are generated manually by labeling each patch pixel as a vessel pixel (white) or a background pixel (black). In some instances, both the patch and the annotation are 40×40 pixels. It is understood that any N×N size (where N is the number of pixels) of patches and annotations can be used to train the network, as long as the patches and annotations have exactly the same size. 687 annotations are used to train the network, and 278 annotations are used for validation. In this example implementation, the segmentation network consists of an APN and a U-Net backbone with a VGG-19 encoder. It is understood that the APN can be replaced by other preprocessing functions or networks, such as a pixel intensity normalization process or manual parameters for contrast enhancement and boundary sharpening. It is also understandable that U-Net can be replaced by other semantic segmentation networks, such as DeeplabV3+. The narrow segmentation network is trained for 400 epochs with a learning rate of 1e-4 and the ADAM optimizer. During training, weight decay (L2) regularization with a coefficient of 1e-3 is used. The loss function is a weighted average of the Tversky loss and the centerline Dice loss. The output of the narrow segmentation neural network 1252 is an N×N segmented image patch, where blood vessel pixels are marked as white and background pixels are marked as black. The segmented image patch is then used for further analysis; for example, the patch can be fed into the patch insertion script 1254, the patch smoothing script 1258, or the narrow quantization module 1256.

[0110] Figure 16 An example process 1500, which can be executed via workflow 1204, is shown for generating enhanced 2D segmented images to determine the occlusion status within a vascular examination area. At box 1502, 2D angiographic image data of the vascular examination area is obtained, and at box 1504, one or more 2D segmented vascular images are generated from the 2D angiographic image data. Box 1504 can execute the process described herein, such as... Figure 3 , 4The process in section 5 or any known technique for generating 2D segmented images can be used. That is, regardless of the technique used to generate 2D segmented vascular images, the stenosis level enhancement of process 1500 can be applied to any 2D segmented image. At box 1506, the 2D angiography image data received from box 1502 is fed to a stenosis localization neural network trained to identify stenosis locations within the vascular examination area. In the illustrated example, box 1506 is performed by a neural network that identifies the centroid coordinates of the stenosis. For each of the identified stenosis, box 1508 uses the stenosis centroid coordinates to crop the original angiography image data into N×N pixel patch images. The patch images from box 1508 are fed to box 1510, where the stenosis segmentation neural network segments each patch image. Then, at box 1512, the segmented patch images from box 1510 are stitched together with the 2D segmented image from box 1504. Then, a smoothing process box 1513 is applied to the stitched 2D segmented image. This smoothing process box smooths the segmented patch image and aligns the segmented patch image with the 2D segmented image to form an enhanced 2D segmented image. At box 1514, the degree of stenosis, stenosis length, or other relevant measurements, such as stenosis eccentricity, are determined for each stenosis location in the enhanced 2D segmented image. In some instances, box 1514 may be applied to the segmented patch image before smoothing. At box 1516, the stenosis locations from box 1506, the enhanced 2D segmented image from box 1513, and the stenosis measurements from box 1514 can then be stored, for example, for generating a three-dimensional (3D) vascular tree model.

[0111] Return to Figure 12 Enhanced 2D segmentation images 1210 can be generated for angiography images captured from different clinical projection angles (e.g., RAO / CRA, LAO / CAU, RAO / AP, or other angles). Two or more enhanced 2D segmentation images 1210 from different projection angles can then be provided to a vascular tree reconstruction module 1212. Using this input, the vascular tree reconstruction module reconstructs a 3D and / or 1D geometric model of the vascular tree. Module 1212 can be... Figure 11B The 3D reconstruction process shown in 1104, or Figure 2 The 3D modeler 208 and 1D modeler 209 are shown. In some instances, module 1212 is based on... Figure 6A and 6B Any process described herein may be used to reconstruct a 3D (and / or optionally 1D) vascular tree geometry model. In other instances, module 1212 may be a machine learning module trained to reconstruct a 3D vascular tree model. Example machine learning modules include convolutional neural networks such as ResNet101 or Inception v3.

[0112] Workflow 1200 also determines functional data, particularly flow data, of the entire vascular tree or each branch of the vascular tree within the vascular examination area by analyzing dynamic angiography image data 1202. Image data 1202 may contain angiography images captured at different times during dye injection and dye flow through the vascular examination area (e.g., as dynamic angiography image data). Dynamic image data 1202 is fed to flow data generator 1216, which can utilize a series of image processing or machine learning algorithms (such as... Figure 11E and 11F (Those in 1107) are used for implementation. In some instances, the flow data generator 1216 can be configured as a physical information deep learning framework or a TIMI frame counter, which is configured to count the number of frames between key anatomical landmarks. In another instance, the automatic flow extractor can apply a series of image processing techniques to calculate the flow rate through the vessel of interest.

[0113] Figure 17A and 17B The combined description illustrates an example process 1400 for extracting flow data across an entire vascular tree or a single branch of interest, as can be performed by the flow data generator 1216. First, an NN 2D segmentation module 1204 is applied to each angiographic image in dynamic image data 1202 to obtain a series of 2D segmented images 1402. It is understood that preprocessing techniques, such as cropping or removing small branches, can be applied to the series of 2D segmented images 1402. Second, in each 2D segmented image, the number of vascular pixels (white) is counted to generate a vascular pixel count array 1402 for each image. Third, the vascular pixel count array 1402 for each image is divided by its maximum value to generate a normalized vascular pixel count array 1402. Fourth, an image processing or machine learning module is then applied to the vascular pixel count array 1402 for each image or the normalized vascular pixel count array 1402 for each image to extract flow data 1404 across the entire tree or a single branch. This module may include, but is not limited to: smoothing high-frequency noise in blood vessel pixel counts using an exponential moving window or a low-pass filter, converting pixel values ​​to length units, converting frames to time units, estimating the radius along the blood vessel centerline using Euclidean distance transformation and central skeletonization, filtering out cardiac motion, performing deformable body registration, calibrating extracted flow data to account for injection timing during the cardiac cycle, and obtaining parameters for the exponential rise and fall of corresponding portions of the pixel count array.

[0114] The flow data generator 1216 outputs flow data. The boundary conditions of the computational fluid dynamics machine learning module 1218 are defined by the output of the flow data generator 1216 and the patient pressure data 1215. The computational fluid dynamics machine learning module 1218 generates occlusion status data 1222 for the vascular examination area, such as flow and pressure estimates, FFR, and other metrics such as iFR, QFR, and microvascular resistance indices. In some cases, the computational fluid dynamics machine learning model 1218 can be a reduced-order model (ROM), such as... Figure 2 Graph theory information or other ROM 213 within. Figure 12 In this model, the ROM is given by a multi-fidelity neural network model 1220 instead of a graph-theoretic information model. The multi-fidelity NN ROM 1220 is obtained using models with different fidelities, such as low-fidelity 0D or 1D models and high-fidelity 3D Navier-Stokes or basic fact data (such as intrusive FFR), to combine the computational efficiency of low-fidelity models with the accuracy of high-fidelity models.

[0115] In the example implementation of the multi-fidelity NN model 1220, training mode 1700 (see...) Figure 18A The system includes the following: 1) inputs to a 3D (i.e., high-fidelity) and 1D (i.e., low-fidelity) representation 1702 of a vascular tree given by a vascular tree reconstruction module 1212; 2) boundary conditions 1704 given by the output (e.g., assimilated flow) of a flow data generator 1216 and patient pressure 1215; 3) a low-fidelity model (LFM) 1706 representing the flow and pressure physical properties in each vascular tree (characterized by its computational efficiency); 4) a high-fidelity model (HFM) 1708 representing the flow and pressure physical properties in each vascular tree (characterized by its accuracy); 5) a feature extraction machine learning method (e.g., implemented by a feature extraction neural network 1710) for obtaining a low-rank representation of the geometry of the vascular tree; and 6) a machine learning method (e.g., implemented by a multi-fidelity neural network 1711) for establishing the relationship between the low-fidelity models and the high-fidelity models and their corresponding representations of the flow and pressure physical properties in each vascular tree. Multifidelity neural networks can be trained on a large amount of synthetic basic fact data, which consists of high-fidelity CFD simulations of flow and pressure through narrowed vessels, narrowed vessel trees, and basic fact data on FFR, iFR, QFR, or other measures of severity of narrowing.

[0116] In some instances, the low-fidelity model 1706 includes a pressure drop model traversing each stenosis in the vascular tree. This pressure drop model may include a reduction in order of the 3-D Navier-Stokes equations using assumptions of steady-state, axisymmetric, and incompressibility, as well as variable separation of axial velocity. The pressure drop model describes the pressure drop dP / dz as a function of flow rate Q and vascular geometry R(z). This model may contain several terms, some of which have adjustable coefficients that can be adjusted using data. An example of such a dP / dz function is given by:

[0117] ,in and It is an adjustable coefficient.

[0118] In some instances, a pressure drop model penetrating a stenosis can be obtained through tensor-based reduction of axial velocity. This model is also a function of flow rate Q and vessel geometry R(z). The model has adjustable coefficients. It is adjusted to suit the complexity of the available data (i,j = 1, … N). )

[0119] In some instances, the feature extraction machine learning method is a convolutional neural network (FENN) 1710, such as an autoencoder for dimensionality reduction. This autoencoder contains an encoder for compressing the input geometry of each vascular tree into latent spatial features, which provide a low-rank representation of the input geometry for each vascular tree. The autoencoder contains a decoder for recovering the input geometry of each vascular tree from the latent spatial features. The autoencoder is trained to learn geometric features of the vascular tree, such as the number of branches, the number of stenosis areas, the severity of stenosis, the distribution of branch diameters, etc. The autoencoder is trained to maximize attention to stenotic regions.

[0120] In some instances, the machine learning method used to establish the relationship between the low-fidelity model and the high-fidelity model is a multi-fidelity neural network 1711 consisting of the following components: 1) an encoder 1712 for compressing hemodynamic data from the low-fidelity model 1706 into a low-fidelity feature map in the latent space; 2) a decoder 1714 for reconstructing the hemodynamic data of the high-fidelity model (e.g., a 3D representation) from the high-fidelity feature map in the latent space; 3) a fully connected neural network (FCNN) 1716 between the encoder 1712 and the decoder 1714 for mapping the relationship between the low-fidelity feature map and the high-fidelity feature map in the latent space; and 4) a skip connection 1713 (only one is shown, but multiple connections may exist) between the encoder 1712 and the decoder 1714 for enhancing the relationship between the high-fidelity feature map and the low-fidelity feature map.

[0121] In some instances, the inputs to a fully connected neural network are: 1) geometric features learned by a Feature Extraction Neural Network (FENN)1710; and 2) low-fidelity feature maps in the latent space. Similarly, the output of a fully connected neural network can be a high-fidelity feature map.

[0122] In the example implementation of the multi-fidelity NN 1220, production mode 1750 ( Figure 18B The system includes the following: 1) receiving 3D (i.e., high-fidelity) and 1D (i.e., low-fidelity) representations of the vascular tree given by the vascular tree reconstruction module 1212 1752; 2) receiving boundary conditions 1754 given by the output (e.g., assimilated flow) of the flow data generator 1216 and the patient pressure 1215; 3) a low-fidelity model (LFM) 1706 representing the flow and pressure physical properties in each vascular tree; 4) a trained feature extraction neural network 1710 for obtaining a low-rank representation of the geometry of the vascular tree; and 5) a trained multi-fidelity neural network 1711 for representing the flow and pressure physical properties in each vascular tree.

[0123] In this example production mode, the output data of the multi-fidelity NN 1711 (labeled as 1718) can be the occlusion status data 1222 of the vascular examination area, such as flow and pressure estimates, FFR, other indicators, such as iFR, QFR, microvascular resistance indicators, etc.

[0124] Figure 19An example procedure 1600 is demonstrated to analyze vascular health by using a multi-fidelity neural network to determine the state of occlusion in a vascular examination area. At box 1602, enhanced 2D segmentation images are obtained, for example, from procedure 1500. At box 1604, these enhanced 2D segmentation images are fed into a vascular tree reconstruction module to generate 3D and / or 1D geometric vascular tree models. At box 1606, the 3D and / or 1D models are then fed into a multi-fidelity reduction model to generate determinations of vascular health, such as determining the state of occlusion or microvascular disease in the vascular examination area.

[0125] Other aspects

[0126] Aspect 1. A computer-implemented method for generating enhanced segmented images for determining the occlusion status within a vascular examination region, the method comprising: receiving, by one or more processors, angiographic image data of a vascular examination region containing a vascular tree, the angiographic image data comprising one or more two-dimensional (2D) angiographic images; applying, by the one or more processors, the angiographic image data to the angiographic image data of a vascular segmentation machine learning model trained to generate segmented images of the vascular tree; applying, by the one or more processors, the angiographic image data of a stenosis machine learning model trained to identify and segment stenosis within the vascular tree to generate segmented patch images; determining, by the one or more processes, the degree of stenosis of each segmented patch image within the vascular tree; stitching, by the one or more processors, each segmented patch image to the segmented images of the vascular tree to form an enhanced segmented image of the vascular tree; and storing the enhanced segmented image and the determined degree of stenosis of each stenosis for reconstructing a three-dimensional (3D) vascular tree model, performing flow extraction, and determining the occlusion status within the vascular examination region.

[0127] Aspect 2. The computer-implemented method according to aspect 1, wherein the angiography-level vessel segmentation machine learning model is a convolutional neural network.

[0128] Aspect 3. The computer-implemented method according to aspect 2, wherein the convolutional neural network has an APN preprocessing neural network and a Deeplabv3+ architecture.

[0129] Aspect 4. The computer-implemented method according to aspect 1, wherein the stenosis machine learning model comprises a stenosis localization machine learning model trained to identify stenosis locations within the vascular tree and a stenosis segmentation machine learning model trained to generate segmented patch images corresponding to each identified stenosis.

[0130] Aspect 5. The computer-implemented method according to aspect 4, wherein the narrow localization machine learning model is a first convolutional neural network, and wherein the narrow segmentation machine learning model is a second convolutional neural network.

[0131] Aspect 6. The computer-implemented method according to aspect 5, wherein the first convolutional neural network is a target detector.

[0132] Aspect 7. The computer-implemented method according to aspect 5, wherein the second convolutional neural network has an APN preprocessing network and a VGG U-Net architecture.

[0133] Aspect 8. The computer-implemented method according to aspect 1, wherein the angiography-level vessel segmentation machine learning model and the stenosis machine learning model each have an angiography processing network (APN) trained to remove ducts, overlapping bone structures and / or other image artifacts from the respective input data images.

[0134] Aspect 9. The computer-implemented method according to aspect 4, wherein the stenosis localization machine learning model is trained to identify the centroid coordinates of each stenosis in angiographic image data.

[0135] Aspect 10. The computer-implemented method according to aspect 4, wherein the stenosis segmentation machine learning model is trained to generate a segmented patch image by classifying blood vessel and background pixels in a cropped patch image of the angiography image data.

[0136] Aspect 11. The computer-implemented method according to aspect 10, wherein the cropped patch image of the angiography image data is given by the centroid coordinates of the identified stenosis and an NxN pixel size.

[0137] Aspect 12. The computer-implemented method according to aspect 1, wherein stitching each generated segmented patch image to the segmented image includes an insertion and smoothing process.

[0138] Aspect 13. The computer-implemented method according to aspect 12, wherein the insertion process includes inserting the segmented patch image into the segmented image using the identified narrow centroid coordinates.

[0139] Aspect 14. The computer-implemented method according to aspect 12, wherein the smoothing process comprises a filling process, an amplification process, a Gaussian filtering process, an erosion process, a reduction process, a thresholding process, and / or a boundary smoothing process performed by the one or more processors based on linear interpolation and a Savitzky-Golay filter.

[0140] Aspect 15. The computer-implemented method according to aspect 1, comprising determining the degree of narrowing for each narrowing location in each segmented patch image.

[0141] Aspect 16. The computer-implemented method according to aspect 15, wherein the degree of narrowing determined for each segmented patch image includes narrowing percentage data, narrowing length data, and narrowing eccentricity data.

[0142] Aspect 17. The computer-implemented method according to aspect 1, further comprising: feeding the enhanced segmented image to a vascular tree geometry model builder; and generating a segmented geometric vascular tree model in the vascular tree geometry model builder.

[0143] Aspect 18. The computer-implemented method according to aspect 17, wherein the segmented geometric vascular tree model is a 3D model.

[0144] Aspect 19. The computer-implemented method according to aspect 17, wherein the segmented geometric vascular tree geometric model is a 1D model.

[0145] Aspect 20. A computer-implemented method according to aspect 1, wherein the angiography image data includes a dynamic angiography image corresponding to the progress of fluid flow within the vascular tree, the method further comprising: applying the dynamic angiography image to a flow data generator configured to extract flow data of the vascular tree; and applying the segmented geometric vascular tree model, the extracted flow data, and pressure data to a reduced-order model (ROM) configured to determine the occlusion state in the vascular examination region.

[0146] Aspect 21. A computer-implemented method according to aspect 20, wherein the ROM for determining the occlusion state in the vascular examination region is a multi-fidelity neural network, the multi-fidelity neural network being configured to: receive, by one or more processors, a segmented geometric vascular tree model represented by a matrix Mi×Ni×4, where Mi is the number of branches in vascular tree i, Ni is the number of points on the centerline of each branch, and 4 is data encoded at each point on the centerline of the branch, specifically its three-dimensional coordinates (x, y, z) and radius r; obtain a low-rank representation of the geometry of the segmented geometric vascular tree model using a feature extraction machine learning method; represent the flow and pressure physical characteristics in each vascular tree using a low-fidelity model; represent the flow and pressure physical characteristics in each vascular tree using a high-fidelity model; and establish a relationship between the low-fidelity model and the high-fidelity model and their corresponding representations of the flow and pressure physical characteristics in each vascular tree using a machine learning method.

[0147] Aspect 22. The computer-implemented method according to aspect 21, wherein the feature extraction machine learning model is a convolutional neural network (FECNN).

[0148] Aspect 23. The computer-implemented method according to aspect 22, wherein the convolutional neural network is an autoencoder for dimensionality reduction.

[0149] Aspect 24. The computer-implemented method according to aspect 23, wherein the autoencoder includes an encoder for compressing the input geometry of each vascular tree into latent spatial features, the latent spatial features providing a low-rank representation of the input geometry of each vascular tree.

[0150] Aspect 25. The computer-implemented method according to aspect 23, wherein the autoencoder contains a decoder for recovering the input geometry of each vascular tree i from the latent spatial features.

[0151] Aspect 26. The computer-implemented method according to aspect 23, wherein the autoencoder is trained to learn geometric features of the vascular tree i, such as the number of branches Mi, the number of stenosis, the severity of stenosis, the distribution of branch diameters, etc.

[0152] Aspect 27. The computer-implemented method according to aspect 21, wherein the low-fidelity model is given by a lumped parameter model, a 1-D linear model of the Navier-Stokes equation, a 1-D nonlinear model of the Navier-Stokes equation, or a coarse-grid 3-D Navier-Stokes model.

[0153] Aspect 28. The computer-implemented method according to aspect 27, wherein the low-fidelity model comprises a pressure drop model through each stenosis in the vascular tree.

[0154] Aspect 29. The computer-implemented method according to aspect 28, wherein the pressure drop model through the narrow passage is obtained by reducing the order of the 3-D Navier-Stokes equations using assumptions of steady state, axisymmetry, and incompressibility, as well as variable separation of axial velocity.

[0155] Aspect 30. The computer-implemented method according to aspect 28, wherein the pressure drop model through the narrow passage is obtained by tensor-based reduction of axial velocity.

[0156] Aspect 31. The computer-implemented method according to aspect 21, wherein the high-fidelity model is given by a fine-grid 3-D Navier-Stokes model of pressure and flow or an invasive FFR assessment of each vascular tree.

[0157] Aspect 32. The computer-implemented method according to aspect 21, wherein the machine learning method for establishing the relationship between the low-fidelity model and the high-fidelity model is a multi-fidelity neural network.

[0158] Aspect 33. A computer-implemented method according to aspect 32, wherein the multi-fidelity neural network comprises: an encoder for compressing hemodynamic data from the low-fidelity model into a low-fidelity feature map in a latent space; a decoder for reconstructing the hemodynamic data of the high-fidelity model from the high-fidelity feature map in the latent space; a fully connected neural network for mapping the relationship between the low-fidelity feature map and the high-fidelity feature map in the latent space; and a skip connection between the encoder and the decoder for enhancing the relationship between the high-fidelity feature map and the low-fidelity feature map.

[0159] Aspect 34. The computer-implemented method according to aspect 33, wherein the input to the fully connected neural network is: geometric features learned by the computer-implemented method according to aspect 3; and the low-fidelity feature map.

[0160] Aspect 35. The computer-implemented method according to aspect 33, wherein the output of the fully connected neural network is: the high-fidelity feature map.

[0161] Throughout this specification, multiple instances can implement components, operations, or structures described as single instances. Although the various operations of one or more methods are shown and described as separate operations, one or more of the operations can be performed simultaneously, and they do not need to be performed in the order shown. Structures and functionalities presented as separate components in the example configurations can be implemented as composite structures or components. Similarly, structures and functionalities presented as single components can be implemented as single components. These, as well as other variations, modifications, additions, and improvements, all fall within the scope of this document.

[0162] Additionally, some embodiments are described herein as including logic or multiple routines, subroutines, applications, or instructions. These can constitute software (e.g., code embodied on a non-transitory machine-readable medium) or hardware. In hardware, routines, etc., are tangible units capable of performing certain operations and can be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., standalone client or server computer systems) or one or more hardware modules of a computer system (e.g., processors or processor groups) can be implemented via software (e.g., applications or application portions, such as...). Figure 2The contrast agent injection system shown is a hardware module configured to operate to perform certain operations as described herein.

[0163] In various embodiments, the hardware module may be implemented mechanically or electronically. For example, the hardware module may include dedicated circuitry or logic (e.g., a dedicated processor such as a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC)) that is permanently configured to perform certain operations. The hardware module may also include programmable logic or circuitry (e.g., encompassed within a dedicated processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be understood that the decision to implement the hardware module mechanically, either in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software), may be driven by cost and time considerations.

[0164] Therefore, the term "hardware module" should be understood to encompass tangible entities, meaning entities that are physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein. Considering embodiments where hardware modules are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware module at any given time. For example, in cases where a hardware module contains a general-purpose processor configured using software, the general-purpose processor can be configured as different hardware modules at different times. Thus, software can configure the processor to, for example, constitute a specific hardware module at one time and different hardware modules at different times.

[0165] Hardware modules can provide information to and receive information from other hardware modules. Therefore, the described hardware modules can be considered communicatively coupled. In the presence of multiple such hardware modules, communication can be achieved through signal transmission connecting the hardware modules (e.g., via appropriate circuitry and buses). In embodiments where multiple hardware modules are configured or instantiated at different times, communication between such hardware modules can be achieved, for example, by storing and retrieving information in a memory structure accessible to the multiple hardware modules. For example, one hardware module can perform an operation and store the output of such operation in a communicatively coupled memory device. Another hardware module can then access this memory device at a later time to retrieve and process the stored output. Hardware modules can also initiate communication with input or output devices and can operate on resources (e.g., collections of information).

[0166] The various operations of the example methods described herein can be performed, at least in part, by one or more processors that are temporarily (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented modules that operate to perform one or more operations or functions. In some example embodiments, the modules referred to herein may include processor-implemented modules.

[0167] Similarly, the methods or routines described herein can be implemented at least in part by a processor. For example, at least some operations of a method can be performed by one or more processors or hardware modules implemented by processors. The performance of certain operations can be distributed among one or more processors, not only residing within a single machine but also deployed across multiple machines. In some example embodiments, one or more processors may reside in a single location (e.g., in a home environment, an office environment, or as a server farm), while in other embodiments, processors may be distributed across multiple locations.

[0168] The performance of certain operations can be distributed across one or more processors, residing not only within a single machine but also deployed across multiple machines. In some example embodiments, one or more processors or processor-implemented modules may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, one or more processors or processor-implemented modules may be distributed across multiple geographic locations.

[0169] Unless otherwise expressly stated, discussions using terms such as “processing,” “computing,” “calculating,” “determining,” “presenting,” and “displaying” in this document may refer to the actions or processes of a machine (e.g., a computer) to manipulate or transform data represented as physical (e.g., electrical, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0170] As used herein, any reference to "an embodiment" or "embodiment" means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The phrase "in an embodiment" appearing in various places in this specification does not necessarily refer to the same embodiment.

[0171] Some embodiments may be described using the expressions “coupled” and “connected” and their derivatives. For example, the term “coupled” may be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact. However, the term “coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. Embodiments are not limited in this context.

[0172] Those skilled in the art will recognize that various modifications, alterations, and combinations can be made with respect to the above embodiments without departing from the scope of the invention, and such modifications, alterations, and combinations will be considered to be within the scope of the concept of the invention.

[0173] Although the invention has been described with reference to specific examples, which are intended to be illustrative and not limiting, it will be apparent to those skilled in the art that changes, additions and / or deletions can be made to the disclosed embodiments without departing from the spirit and scope of the invention.

[0174] The foregoing description is provided for clarity of understanding; and should not be construed as an unnecessary limitation, as modifications within the scope of the invention will be readily apparent to those skilled in the art.

Claims

1. A computer-implemented method for generating enhanced segmented images to determine the state of occlusion within a vascular examination area, the method comprising: One or more processors receive angiographic image data of a vascular examination area containing a vascular tree, the angiographic image data comprising one or more two-dimensional (2D) angiographic images; The angiography image data is applied by the one or more processors to an angiography-level vascular segmentation machine learning model trained to generate segmented images of the vascular tree; The angiography image data is applied by the one or more processors to a stenosis machine learning model trained to identify and segment stenosis within the vascular tree to generate segmented patch images; The degree of stenosis in each segmented patch image within the vascular tree is determined by one or more of the processes described above; Each segmented patch image is stitched together by the one or more processors into the segmented image of the vascular tree to form an enhanced segmented image of the vascular tree; as well as The enhanced segmented images and the determined degree of stenosis for each stenosis are stored for reconstructing a three-dimensional (3D) vascular tree model, performing flow extraction, and determining the occlusion status in the vascular examination area.

2. The computer-implemented method according to claim 1, wherein the angiography-level vessel segmentation machine learning model is a convolutional neural network.

3. The computer-implemented method according to claim 2, wherein the convolutional neural network has an APN preprocessing neural network and a Deeplabv3+ architecture.

4. The computer-implemented method of claim 1, wherein the stenosis machine learning model comprises a stenosis localization machine learning model trained to identify stenosis locations within the vascular tree and a stenosis segmentation machine learning model trained to generate segmented patch images corresponding to each identified stenosis.

5. The computer-implemented method of claim 4, wherein the narrow localization machine learning model is a first convolutional neural network, and wherein the narrow segmentation machine learning model is a second convolutional neural network.

6. The computer-implemented method of claim 5, wherein the first convolutional neural network is a target detector.

7. The computer-implemented method of claim 5, wherein the second convolutional neural network has an APN preprocessing network and a VGG U-Net architecture.

8. The computer-implemented method of claim 1, wherein the angiography-level vessel segmentation machine learning model and the stenosis machine learning model each have an angiography processing network (APN) trained to remove ducts, overlapping bone structures and / or other image artifacts from the respective input data images.

9. The computer-implemented method of claim 4, wherein the stenosis localization machine learning model is trained to identify the centroid coordinates of each stenosis in the angiographic image data.

10. The computer-implemented method of claim 4, wherein the stenosis segmentation machine learning model is trained to generate a segmented patch image by classifying blood vessel and background pixels in a cropped patch image of the angiography image data.

11. The computer-implemented method of claim 10, wherein the cropped patch image of the angiography image data is given by the centroid coordinates of the identified stenosis and an NxN pixel size.

12. The computer-implemented method of claim 1, wherein stitching each generated segmented patch image to the segmented image includes an insertion and smoothing process.

13. The computer-implemented method of claim 12, wherein the insertion process comprises inserting the segmented patch image into the segmented image using the identified narrow centroid coordinates.

14. The computer-implemented method of claim 12, wherein the smoothing process comprises a filling process, an amplification process, a Gaussian filtering process, an erosion process, a reduction process, a thresholding process, and / or a boundary smoothing process performed by the one or more processors based on linear interpolation and a Savitzky-Golay filter.

15. The computer-implemented method of claim 1, comprising determining the degree of narrowing for each narrow location in each segmented patch image.

16. The computer-implemented method of claim 15, wherein the degree of narrowing determined for each segmented patch image includes narrowing percentage data, narrowing length data, and narrowing eccentricity data.

17. The computer-implemented method according to claim 1, further comprising: The enhanced segmented image is fed into the vascular tree geometry model builder; and The segmented geometric vascular tree model is generated in the vascular tree geometry model builder.

18. The computer-implemented method of claim 17, wherein the segmented geometric vascular tree model is a 3D model.

19. The computer-implemented method of claim 17, wherein the segmented geometrical vascular tree geometric model is a 1D model.

20. The computer-implemented method of claim 1, wherein the angiographic image data comprises dynamic angiographic images corresponding to the progression of fluid flow within the vascular tree, the method further comprising: The dynamic angiography images are applied to a flow data generator configured to extract flow data from the vascular tree; and The segmented geometric vascular tree model, the extracted flow data, and the pressure data are applied to a reduced-order model (ROM) configured to determine the occlusion state in the vascular examination region.

21. The computer-implemented method of claim 20, wherein the ROM for determining the occlusion state in the vascular examination region is a multi-fidelity neural network, the multi-fidelity neural network being configured to: The segmented geometric vascular tree model, represented by a matrix Mi×Ni×4, is received by one or more processors, where Mi is the number of branches in vascular tree i, Ni is the number of points on the center line of each branch, and 4 is the data encoded at each point on the center line of the branch, specifically its three-dimensional coordinates (x, y, z) and radius r. A low-rank representation of the geometry of the segmented geometric vascular tree model is obtained using a feature extraction machine learning method. Low-fidelity models are used to represent the flow and pressure physical properties in each vascular tree; High-fidelity models are used to represent the flow and pressure physical properties within each vascular tree; and Machine learning methods are used to establish the relationship between the low-fidelity model and the high-fidelity model, as well as their corresponding representations of flow and pressure physical properties in each vascular tree.

22. The computer-implemented method of claim 21, wherein the feature extraction machine learning model is a convolutional neural network (FECNN).

23. The computer-implemented method of claim 22, wherein the convolutional neural network is an autoencoder for dimensionality reduction.

24. The computer-implemented method of claim 23, wherein the autoencoder includes an encoder for compressing the input geometry of each vascular tree into latent spatial features, the latent spatial features providing a low-rank representation of the input geometry of each vascular tree.

25. The computer-implemented method of claim 23, wherein the autoencoder includes a decoder for recovering the input geometry of each vascular tree i from the latent spatial features.

26. The computer-implemented method of claim 23, wherein the autoencoder is trained to learn geometric features of the vascular tree i, such as the number of branches Mi, the number of stenosis, the severity of stenosis, the distribution of branch diameters, etc.

27. The computer-implemented method of claim 21, wherein the low-fidelity model is given by a lumped parameter model, a 1-D linear model of the Navier-Stokes equation, a 1-D nonlinear model of the Navier-Stokes equation, or a coarse-grid 3-D Navier-Stokes model.

28. The computer-implemented method of claim 27, wherein the low-fidelity model comprises a pressure drop model throughout each stenosis in the vascular tree.

29. The computer-implemented method of claim 28, wherein the pressure drop model through the narrow passage is obtained by reducing the order of the 3-D Navier-Stokes equations using assumptions of steady state, axisymmetry, and incompressibility, as well as variable separation of axial velocity.

30. The computer-implemented method of claim 28, wherein the pressure drop model through the narrow passage is obtained by tensor-based reduction of axial velocity.

31. The computer-implemented method of claim 21, wherein the high-fidelity model is given by a fine-grid 3-D Navier-Stokes model of pressure and flow or an invasive FFR assessment of each vascular tree.

32. The computer-implemented method of claim 21, wherein the machine learning method for establishing the relationship between the low-fidelity model and the high-fidelity model is a multi-fidelity neural network.

33. The computer-implemented method of claim 32, wherein the multi-fidelity neural network comprises the following components composition: An encoder for compressing hemodynamic data from the low-fidelity model into a low-fidelity feature map in the latent space; A decoder for reconstructing the hemodynamic data of the high-fidelity model from a high-fidelity feature map in the latent space; A fully connected neural network, wherein the fully connected neural network is used to map the relationship between low-fidelity feature maps and high-fidelity feature maps in a latent space; and Skip connections between the encoder and decoder, which are used to enhance the relationship between high-fidelity feature maps and low-fidelity feature maps.

34. The computer-implemented method of claim 33, wherein the input to the fully connected neural network is: Geometric features learned by the computer-implemented method according to claim 3; and The low-fidelity feature map.

35. The computer-implemented method of claim 33, wherein the output of the fully connected neural network is: The high-fidelity feature map.

Images