Method and system for evaluating functionally significant vascular occlusion based on machine learning

Abstract

A method and system are provided for evaluating a cardiovascular occlusion from a volume image dataset. The volume image dataset is analyzed to extract data representing an axial trajectory of the cardiovascular. Multi-planar reformation (MPR) images and data representing the axial trajectory of the cardiovascular are generated. The MPR images are supplied as an input to a first machine learning network that outputs feature data characterizing a plurality of features of the cardiovascular along the axial trajectory of the cardiovascular. Additional data characterizing at least one additional feature of the cardiovascular along the axial trajectory of the cardiovascular is generated. The data output by the first machine learning network and the additional data are input to a second machine learning network that outputs data characterizing the severity of an anatomical lesion of the cardiovascular.

Description

Technical Field

[0001] This application relates to the technical field of medical images, particularly computer tomography angiography images, but can be applied to any field where it is necessary to quantify the flow rate in occluded or partially occluded conduits, such as in non-destructive testing applications.

Background Art

[0002] Coronary artery disease (CAD) is one of the leading causes of death worldwide. CAD generally refers to a condition with narrowed or occluded blood vessels, which can reduce or eliminate blood supply to the distal part from the stenosis, reduce oxygen supply to the myocardium, and may cause, for example, ischemia and chest pain (angina). Narrowing of blood vessels is called stenosis, which is caused by arteriosclerosis. Arteriosclerosis is the accumulation of fat, cholesterol, and other substances on the blood vessel wall (plaque) (see Figure 1). Atherosclerotic plaques can be classified into calcified plaques, soft plaques, and mixed plaques (plaques containing calcified and non-calcified components) according to their components. Such non-calcified components include extracellular matrix, smooth muscle cells, macrophages, foam cells, lipids, and fibrous tissue. Calcified plaques are considered stable, and the amount of calcified plaques in the coronary arteries is a strong predictor of cardiovascular events. Different from calcified plaques, non-calcified plaques and mixed plaques are considered unstable and prone to rupture. Rupture of plaques, when occurring in the coronary arteries, may lead to serious acute events such as stroke and heart attack. A heart attack can result in myocardial infarction, which causes irreversible damage to the myocardium. Since the patient management strategy varies depending on the type of plaque and the degree of stenosis, it is important to detect and characterize coronary artery plaques and the degree of stenosis.

[0003] In addition to the degree of stenosis (anatomical stenosis), another important aspect in the prevention and treatment of CAD is the functional evaluation of such anatomically narrowed or occluded blood vessels.

[0004] Currently, X-ray angiography is an imaging diagnostic method used in a catheterization laboratory during the minimally invasive surgical treatment of stenosed coronary arteries, also known as percutaneous coronary intervention (PCI). During PCI, the (interventional) cardiologist inserts a deflated balloon or other device attached to a catheter through an artery, either the femoral artery in the groin or the radial artery, through the blood vessels to the site of arterial occlusion. X-ray images are used to guide the insertion of the catheter. In PCI, typically, the balloon is inflated to open the artery with the aim of restoring unobstructed blood flow. To keep the artery open, a stent or scaffold may be placed at the occlusion site. For example, in the case of moderate coronary anatomical lesions (defined as 30 - 70% lumen stenosis), it is not always clear whether the stenosis poses a risk to the patient and whether treatment is necessary. If the severity of the stenosis is overestimated, a treatment that turns out to be unnecessary may be performed, and as a result, the patient may be exposed to unnecessary risks. On the other hand, if the severity of the stenosis is underestimated, there is a risk that the patient may be left untreated even though the stenosis is actually severe and obstructs blood flow to the myocardium. In particular, in these situations, it is desirable to perform additional functional evaluations to assist in making appropriate decisions.

[0005] Fractional Flow Reserve (FFR) has been increasingly used over the past 10 to 15 years as a method for identifying coronary artery lesions for which PCI can be expected to be effective and effectively targeting them. FFR evaluates the pressure difference across a coronary artery stenosis to determine the likelihood that the stenosis is interfering with oxygen supply to the myocardium. In the characterization of FFR, a pressure transducer wire is typically inserted percutaneously into the coronary artery, and the pressures distal and proximal to the lesion are measured. This evaluation is performed in a catheterization laboratory. Since myocardial blood flow is proportional to myocardial perfusion pressure during maximal hyperemia, FFR is optimally performed in a hyperemic state. For this reason, as described in Non-Patent Document 1, FFR provides a quantitative assessment of the functional severity of coronary artery lesions. The guidelines of Non-Patent Documents 2 and 3 recommend the use of FFR in patients with moderate (30 - 70%) coronary stenosis, but more than 90% of the procedures for selecting patients for percutaneous coronary intervention still use only visual assessment of x-ray coronary angiography, regardless of the confirmation by quantitative coronary analysis (QCA) (Non-Patent Document 4). However, FFR has several drawbacks. For example, the characterization of FFR may require the additional cost of a pressure wire that can only be used once. Furthermore, the characterization of FFR may require an invasive catheterization method that involves associated costs and procedural time. Also, the injection of drugs (adenosine or papaverine) may be required to induce (maximal) hyperemia, which is an additional burden on the patient.

[0006] Coronary computed tomography (CT) angiography (CCTA) is a non-invasive imaging diagnostic method for the anatomical evaluation of coronary arteries, but it does not evaluate the functional significance of coronary artery lesions. Because of its significantly high negative predictive value and non-invasiveness, the main strength of CCTA lies in its excellent ability to rule out CAD. CCTA can reliably rule out the presence of significant coronary artery disease, but many of the severe stenoses seen on CCTA do not cause blood flow limitation. This possibility of false-positive results has raised concerns that clinically unnecessary coronary revascularization may be performed due to the widespread use of CCTA. This lack of specificity of CCTA is one of the main limitations of CCTA when judging the hemodynamic significance of CAD (Non-Patent Document 5). As a result, CCTA may lead to unnecessary interventions for patients, pose additional risks to patients, and generate unnecessary medical costs.

[0007] Non-Patent Document 6 and Patent Document 1 describe a non-invasive method (FFRCT) for quantifying FFR from CCTA. This technique uses computational fluid dynamics (CFD) applied to CCTA after semi-automatic segmentation of the coronary artery tree, including a part of the ascending aorta covering the regions where both the left and right coronary arteries originate. Blood is modeled as an incompressible Newtonian fluid using the Navier-Stokes equations and solved using the finite element method of a parallel supercomputer according to appropriate initial and boundary conditions to simulate the three-dimensional (3D) blood flow and pressure in the coronary arteries. FFRCT is modeled for the state of adenosine-induced hyperemia without adenosine injection. This process is computationally complex and time-consuming, and may take several hours, and is highly dependent on the 3D anatomical coronary artery model that is the result of segmentation, so it has the same limitations as above.

[0008] In addition to the development of CFD-based FFR prediction methods, an approach has emerged that correlates quantitative metrics obtained from CCTA with measured FFR values. These clinical metrics assess the coronary arteries, for example, by the trans-luminal attenuation gradient (Non-Patent Document 40; Non-Patent Document 41) or plaque volume (Non-Patent Document 42; Non-Patent Document 43), or characterize the coronary arteries by quantifying stenosis (Non-Patent Document 44; Non-Patent Document 43) or contrast density difference (Non-Patent Document 45; Non-Patent Document 46). The mathematical simplicity and intuitive design of the calculated metrics enable these interpretations, but modeling the complex relationship between FFR on CCTA and coronary artery characteristics limits these capabilities. Therefore, to refine FFR prediction using clinical metrics, machine learning classifiers that combine multiple metrics have been employed (Non-Patent Document 47; Itu et al., 2016; Non-Patent Document 48; Non-Patent Document 43; Non-Patent Document 49). This has significantly improved performance compared to that of a single metric. However, these metric-based efforts have the same drawbacks as CFD-based methods. That is, accurate segmentation of the coronary artery lumen is required to calculate the metrics, which can be very difficult, especially when pathology is present (Non-Patent Document 50). Since these methods typically use an automatic segmentation method as a starting point, significant manual operations are periodically required for errors in automatic segmentation.

Prior Art Documents

Patent Documents

[0009]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

[0010] [Non-Patent Document 1] Pijls et al., "Measurement of the Coronary Flow Reserve Ratio to Evaluate the Functional Severity of Coronary Stenosis", N Engl J Med 1996, 334:1703-1708 [Non-Patent Document 2] Guidelines of the European Society of Cardiology (ESC) [Non-Patent Document 3] Guidelines of the American College of Cardiology / American Heart Association (ACC / AHA) [Non-Patent Document 4] Kleiman et al., "Putting it all together: integration of physiology and anatomy in cardiac catheterization", J Am Coll Cardiol. 2011;58:1219-1221 [Non-Patent Document 5] Meijboo et al., "Comprehensive Assessment of Coronary Stenosis: Correlation of Fractional Flow Reserve with Conventional Coronary Angiography and Computed Tomography Coronary Angiography in Patients with Stable Angina Pectoris", Journal of the American College of Cardiology 52 (8)(2008)636-643 [Non-Patent Document 6] Taylor et al., "Non-invasive quantification of stroke volume by computational fluid dynamics applied to cardiac computed tomography," Journal of the American College of Cardiology, Vol. 61, No. 22, 2013 [Non-Patent Document 7] Litjens et al., "A survey on deep learning in medical image analysis," Med Image Anal. 2017 Dec;42:60-88 [Non-Patent Document 8] Suganyadevi et al., "A review of deep learning in medical image analysis," International journal of multimedia information retrieval vol. 11,1 (2022) [Non-Patent Document 9] Varoquaux et al., "Machine learning in medical imaging: methodological failures and recommendations for the future," NPJ digital medicine vol. 5,1 48. 12 Apr. 2022 [Non-Patent Document 10] Abbara et al., "SCCT guidelines for the performance and acquisition of coronary computed tomography angiography: a report of the Society of Cardiovascular Computed Tomography (NASCI) - recommended cardiovascular computed tomography guidelines committee," J Cardiovasc Comput Tomogr. 2016 Nov -Dec;10(6):435-449 [Non-Patent Document 11] Metz et al., "Semi-automatic coronary centerline extraction in computed tomography angiography data," proceedings / IEEE International Symposium on Biomedical Imaging: from nano to macro, May 2007 [Non-Patent Document 12] Wolterink et al., "Coronary centerline extraction in cardiac CT angiography using a CNN-based direction classifier," Med Image Anal. 2019 Jan;51:46-60 [Non-Patent Document 13] In addition to Austen, "Reporting System for Patients Undergoing Evaluation of Coronary Artery Disease. Report of the Ad Hoc Committee on Grading of Coronary Artery Disease, Council on Cardiovascular Surgery, American Heart Association", Circulation 51, 5 - 40. 1975

Non - Patent Document 14

Non - Patent Document 15

Non - Patent Document 16

Non - Patent Document 17

Non - Patent Document 18

Non - Patent Document 19

Non-Patent Document 27

Non-Patent Document 28

Non-Patent Document 29

Non-Patent Document 30

Non-Patent Document 31

Non-Patent Document 32

Non-Patent Document 33

Non-Patent Document 34

Non-Patent Document 35

Non-Patent Document 36

Non-Patent Document 37

Non-Patent Document 38

Non-Patent Document 39

Non-Patent Document 40

Non-Patent Document 41

Non-Patent Document 42

Non-Patent Document 43

Non-Patent Document 44

Non-Patent Document 45

Non-Patent Document 46

Non-Patent Document 47

Non-Patent Document 48

Non-Patent Document 49

Non-Patent Document 50

Non-Patent Document 51

Summary of the Invention

Problems to be Solved by the Invention

[0011] Therefore, it is necessary to obtain coronary lesion parameters (such as plaque type, severity of anatomical lesions, severity of functional coronary lesions) without depending on the detailed morphology of the coronary artery system. Means for Solving the Problems

[0012] According to an aspect of the present specification, a method for evaluating vascular occlusion by deep learning-based analysis of volume image data is provided.

[0013] In previous deep learning studies (Patent Documents 2 and 3), several methods for evaluating the functional severity of vascular occlusion by extracting the myocardium and / or MPR of the artery of interest focusing on the region of interest were disclosed.

[0014] In the present application, a new deep learning method and system are provided that use a convolutional neural network (CNN) or variational autoencoder to extract additional features or characteristics along the coronary vasculature. Further, the features or characteristics can be directly extracted from the coronary centerline tree. Such features indicate, for each coronary centerline point, whether it is in the aorta or a collateral and whether there is a branch at that location. These features can be used in combination with other extracted features to evaluate vascular occlusion. For this purpose, a second network is trained to perform both the regression of FFR values, FFR drops, pullback FFR, and the classification of the functional significance of arterial occlusions.

[0015] In an embodiment of the present specification, a method for evaluating the occlusion of a patient's coronary vasculature is obtaining a volume image dataset of the coronary vasculature (e.g., CCTA image data) of a coronary vessel (such as a coronary artery or coronary artery tree); analyzing the volume image dataset to extract data representing the axial trajectory of the coronary vasculature; generating multi-planar reformatted (MPR) images based on the volume image dataset and the data representing the axial trajectory of the coronary vasculature; feeding the MPR images as inputs to a first machine learning network that outputs feature data characterizing a plurality of features of the coronary vasculature along the axial trajectory of the coronary vasculature when the MPR images are provided; generating additional data characterizing at least one additional feature of the coronary vasculature along the axial trajectory of the coronary vasculature by a different analysis separate from the first machine learning network; and Supplying, as input data, the data and the additional data output by the first machine learning network to a second machine learning network that outputs data characterizing the severity of the anatomical lesion of the target blood vessel when the input data is provided comprising.

[0016] This method may further comprise the step of displaying or outputting the data characterizing the severity of the anatomical lesion of the target blood vessel.

[0017] This additional data may be generated from the analysis of the MPR images and / or the analysis of the volume image dataset and / or the coronary centerline tree obtained from the volume image dataset.

[0018] This additional data may characterize at least one of the collateral branches and bifurcations along the axial trajectory of the target blood vessel and / or at least one of the soft plaque area, mixed plaque area or other specific features along the axial trajectory of the target blood vessel.

[0019] This additional data may further characterize the local part of the myocardium related to the target blood vessel.

[0020] As a refinement, the data output by the second machine learning network includes the fractional flow reserve (FFR) value of the entire target blood vessel, and the second machine learning network is advantageously trained by supervised learning using training data including reference annotations based on the measurement of FFR values of a plurality of patients.

[0021] For further refinement, the data output by the second machine learning network includes the coronary blood flow reserve ratio (FFR) value of the center line points along the coronary blood vessel, and the second machine learning network is trained by supervised learning using training data including reference annotations based on the measurement of FFR values associated with the vascular center line points of a plurality of patients.

[0022] Furthermore, for further refinement, the data output by the second machine learning network represents a prediction of the presence of a functionally significant stenosis, and the second machine learning network is trained by supervised learning using training data including reference annotations representing the presence of functionally significant stenoses in a plurality of patients.

[0023] In one embodiment, the plurality of features characterized by the feature data output by the first machine learning network includes at least one feature related to the lumen characteristics (such as lumen area and / or lumen attenuation) of the coronary blood vessel along the axial trajectory of the coronary blood vessel.

[0024] For refinement, the plurality of features characterized by the feature data output by the first machine learning network includes at least one feature related to the plaque characteristics (such as calcium plaque area, soft plaque area, mixed plaque area) of the coronary blood vessel along the axial trajectory of the coronary blood vessel.

[0025] It is advantageous for the first machine learning network to comprise a convolutional neural network trained using training data including reference annotations for the plurality of features characterized by the feature data output by the first machine learning network.

[0026] The reference annotations may be derived by manual segmentation and / or automatic segmentation of the corresponding volume image data.

[0027] The second machine learning network advantageously comprises a convolutional neural network trained using training data including volume image data and corresponding reference annotations for the output data characterizing the severity of the cardiovascular anatomical lesion.

[0028] The reference annotation may be derived by manual segmentation and / or automatic segmentation of the corresponding volume image data.

[0029] The convolutional neural network of the second machine learning system may include a regression head that outputs a fractional flow reserve (FFR) value.

[0030] In refinement, the convolutional neural network of the second machine learning system further includes an accumulator that outputs a fractional flow reserve (FFR) value for a centerline point along the cardiovascular.

[0031] The convolutional neural network of the second machine learning system may include a classification head that outputs data representing a prediction of the presence of a functionally significant stenosis.

[0032] According to one aspect, embodiments herein are also related to a system for evaluating an occlusion of a patient's cardiovascular, the system comprising at least one processor configured to perform some or all of the operations of the method according to embodiments herein when executing program instructions stored in a memory.

[0033] The system advantageously comprises an image acquisition subsystem configured to acquire the volume image data set and / or a display subsystem configured to display the data characterizing the severity of the cardiovascular anatomical lesion.

[0034] Still other variations are contemplated. For example, an embodiment includes generating first feature data characterizing the presence of zero or more branches or side branches along the axial trajectory of the coronary vasculature, which is supplied to the first machine learning network.

[0035] In another embodiment, the additional feature data and / or part or all of the MRP image are adjusted based on the simulated or planned treatment of the coronary vasculature.

[0036] In a further embodiment, it is advantageous for the first machine learning network to be configured to output a plurality of latent space encodings characterizing the features of the coronary vasculature along the axial trajectory of the coronary vasculature when the MPR image is provided, and / or additional feature data characterizing additional features of the coronary vasculature along the axial trajectory of the coronary vasculature. The plurality of latent space encodings and / or the additional feature data output by the first machine learning network may be supplied to a second machine learning network that outputs data characterizing the FFR pullback of the coronary vasculature considering the simulated or planned treatment of the coronary vasculature when input data is provided.

[0037] Embodiments may also provide a method and system for extracting a coronary tree from volumetric image data of a patient's coronary vasculature, which may include one, some, or all of the following operations: Obtaining the volumetric image data set of the coronary vasculature; Tracking a plurality of seed points within the image data set; Using a plurality of seed points to extract an initial representation of the coronary tree within the image data set; Inputting the initial representation of the coronary tree into a first ensemble of graph convolutional neural networks to generate a refined representation of the coronary tree; An operation of using a second ensemble of graph convolutional neural networks to generate labels for segments of the refined representation of the coronary artery tree.

[0038] Other aspects are described and claimed below.

[0039] The features of the present disclosure and the advantages obtained therefrom will become more apparent from the following description of non-limiting embodiments shown in the accompanying drawings described below.

Brief Description of the Drawings

[0040]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5a

Figure 5b

Figure 5c

Figure 5d

Figure 5e

Figure 6

Figure 7a

Figure 7b

Figure 7c

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19a

Figure 19b

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35a

Figure 35b

Figure 35c

Figure 35d

Figure 36

Figure 37

Mode for Carrying Out the Invention

[0041] The term "invisible" as used throughout refers to items that were not used during the training phase. An item in this context means a volume image, a reference value, a feature, and / or other things used during the training phase to train a machine learning model. Instead, invisible features, images, shapes, and other invisible items refer to aspects of a patient or object of interest being analyzed during the prediction phase of an operation.

[0042] The term "FFR value" refers to the FFR value at a specific location within a blood vessel. If the FFR value is used without reference to a location (centerline location), it can refer to the FFR value at the most distal location within the relevant blood vessel.

[0043] The term "FFR pullback" or "FFR pullback graph" refers to the FFR values along the axial trajectory of a relevant blood vessel, as shown, for example, by 1203 in Figure 12 or 2610 in Figure 26.

[0044] The term "FFR drop" means the attenuation of the FFR value along the axial trajectory from the proximal end to the distal end of a relevant blood vessel. The steepness of such attenuation makes it possible to distinguish between local coronary artery disease and chronic coronary artery disease. Local coronary artery disease can be defined as a rapid pressure drop (FFR drop) of the FFR pullback within a relatively short blood vessel segment. On the other hand, chronic coronary artery disease is defined as a gradual pressure loss (FFR drop) along the axial trajectory of the blood vessel, where there is no significant rapid pressure drop at any position along the blood vessel.

[0045] A focal lesion is a local occlusion that can be treated by using a balloon that can dilate the stenosis and then place a stent or scaffold there. Diffuse lesions require different treatment approaches and need to be distinguished from focal lesions to prevent unnecessary costs, patient risks, and patient discomfort due to suboptimal treatment decisions. Thus, using an FFR pullback, it is possible to determine whether there is a focal or diffuse lesion in the blood vessel based on the shape of the virtual pullback.

[0046] Throughout this specification, terms common in the field of machine learning / deep learning are used. For a detailed explanation of these terms, Non-Patent Document 7, Non-Patent Document 8, and Non-Patent Document 9 are referred to, and all of them are incorporated herein by reference.

[0047] The present application relates to a machine learning method and system for evaluating functionally significant vascular occlusions of one or more blood vessels of a target organ based on a contrast-enhanced volume image dataset. Machine learning is a subfield of computer science that "gives a computer the ability to learn without being explicitly programmed." Machine learning, which has evolved from research in pattern recognition and computational learning theory in artificial intelligence, seeks the study and construction of algorithms that can learn from data and make predictions. Such algorithms build models from sample inputs and perform data-driven predictions or decisions, eliminating the need to strictly follow static program instructions. Machine learning is employed in various computing tasks where the design and programming of explicit algorithms are infeasible. Given a dataset of images with known class labels, a machine learning system can predict the class labels of new images. Additionally, machine learning can also be performed using machine learning algorithms in unsupervised learning, which analyzes and clusters unlabeled datasets. These algorithms detect hidden patterns or data groups without the need for human intervention. At a high level, machine learning can be divided into two phases: 1) a training phase in which a model is trained to learn a specific function of a task (e.g., FFR prediction); and 2) a test / validation phase in which this trained model is deployed to unseen data to perform the task (e.g., prediction of FFR).

[0048] In an embodiment, the target organ can be the coronary artery or blood vessels, and in some cases, the heart or a part thereof. A functionally significant vascular occlusion (also referred to as stenosis or lesion) is a hemodynamically significant occlusion of a blood vessel, and with respect to the coronary artery, a coronary artery occlusion that impedes oxygen supply to the myocardium and has the potential to cause angina symptoms is defined. The coronary flow reserve ratio is a hemodynamic index for evaluating a functionally significant coronary artery occlusion. In addition to the coronary flow reserve ratio, other hemodynamic indices can also be used to evaluate a functionally significant coronary artery occlusion, such as coronary artery flow reserve, instantaneous wave-free ratio, congestive myocardial perfusion, microcirculation resistance index, and pressure drop along the coronary artery.

[0049] Embodiments of the present application use machine learning to determine coronary artery parameters related to CAD, such as one or more vascular occlusions, from CCTA data. Machine learning determines coronary artery parameters related to CAD, such as the functional significance of one or more vascular occlusions, from a CCTA data set. Machine learning is a subfield of computer science that "gives a computer the ability to learn without being explicitly programmed." Developed from research in pattern recognition and computational learning theory in artificial intelligence, machine learning explores the research and construction of algorithms that can learn from data and make predictions. Such algorithms build a model from sample inputs and perform data-driven predictions or decisions, eliminating the need to strictly follow static program instructions. Machine learning is employed in various computing tasks where explicit algorithm design and programming are infeasible.

[0050] Figures 2, 17, 23, and 28 each show a flowchart illustrating operations according to embodiments of the present application. These operations employ an imaging system capable of acquiring and processing CCTA datasets of organs (or portions thereof) or other objects of interest. The operations of Figures 2, 17, 23, and 28 (and the operations of other methods, algorithms, and processes described herein) are implemented by one or more processors when executing program instructions. The one or more processors may be implemented on various computing devices such as smartphones, tablet devices, laptop computers, desktop computers, workstations, remote servers, cloud servers, and medical networks. Alternatively, the one or more processors may be distributed across one or more individual computing devices such that some operations are executed by one computing device and the remaining operations are executed by one or more other computing devices.

[0051] Figure 3 is a functional block diagram of an example of a CT system that operates according to commands from a user interface module 301 and provides data to a data analysis module 303. A clinician or other user acquires at least one CT image of a patient to obtain CCTA image data of a volume of interest (e.g., the coronary arteries surrounding the patient's heart). The CCTA image data can be stored in DICOM (Digital Imaging and Communications in Medicine) format on a hard disk, a PACS server, a network server, or a cloud server. The data analysis module 303 may be implemented by a personal computer, a workstation, or other computer processing system. The data analysis module 303 processes the acquired CCTA image data of the CT system 302 to generate, for example, a quantification of coronary artery analysis.

[0052] The user interface module 301 interacts with the user and communicates with the data analysis module 303. The user interface module 301 can include various types of input and output devices such as a display screen for visual output, a touch screen for touch input, a mouse pointer or other pointing device for input, a microphone for voice input, a speaker for audio output, a keyboard and / or keypad for input. Module 304 provides an assessment of a patient's significant coronary artery stenosis. To assess the functional significance of the stenosis of the patient's relevant blood vessels, module 304 is configured to directly apply the deep learning described in this application to the raw CCTA image data acquired by the system. To enable robust training with limited data, this task can be split using two consecutive networks: an arterial characterization network (Figure 2, 204 or Figure 17, 1704) and a stenosis assessment network (Figure 2, 206) or an FFR pullback network (Figure 17, 1706). The preprocessing of the arterial characterization network (Figure 2, 202 or Figure 17, 1702) extracts data representing the axial trajectory of the relevant blood vessels from the CCTA image data of the relevant blood vessels, and generates a multi-planar reconstruction (MPR) of the relevant blood vessels from the extracted axial trajectory and the image data (Figure 2, 203 or Figure 17, 1703). The MPR can be represented by a volume (3-dimensional or "3D") MPR image of the relevant blood vessels and / or a two-dimensional (or "2D") MPR image of the relevant blood vessels.

[0053] The operations of Figure 2, Figure 17, Figure 23 or Figure 28 can be executed by software code incorporated in a computer product (e.g., an optical disk, or other forms of persistent memory such as a USB drive, a network server, a cloud server). The software code can be directly loaded into the memory of the data processing system to execute the operations of Figure 2, Figure 17, Figure 23 or Figure 28.

[0054] In this example, it is assumed that the imaging system acquires and stores at least one CCTA dataset of the coronary vessels. Any imaging device capable of providing a CT scan can be used for this purpose.

[0055] This application is particularly advantageous in the analysis of coronary artery lesion parameters based on CCTA datasets, and this will mainly be disclosed with respect to this field (in particular, patient classification).

[0056] Embodiments of this application are disclosed with reference to FIG. 2. The steps shown in FIG. 2 can, of course, be executed in any logical order and can be partially omitted.

[0057] In step 201 of FIG. 2, a CCTA image dataset is obtained. Such a CCTA image dataset can represent a volumetric CCTA image dataset such as a single contrast-enhanced CCTA dataset. This CCTA dataset can be obtained from a digital storage database such as a Picture Archiving and Communication System (PACS) or a Vendor Neutral Archive (VNA), a local digital storage database, a cloud database, or directly from a CT imaging device. This CCTA dataset can be obtained by a CT imaging operation that injects a contrast agent to enhance the relevant blood vessels (e.g., coronary arteries or blood vessels). At least one purpose of CCTA is to inject an exogenous contrast agent, usually by intravenous injection into the cubital fossa vein, to identify the anatomical structures of the heart and coronary arteries in order to highlight the anatomical structures of the heart and / or coronary arteries during imaging. According to the Society of Cardiovascular Computed Tomography Guidelines regarding the implementation and acquisition of CCTA described in Non-Patent Document 10, the injection of the contrast agent is timed so that the coronary artery system contains sufficient contrast agent and the coronary artery lumen can be clearly distinguished from the surrounding soft tissues. Thereby, the physician can evaluate lumen stenosis and coronary artery stenosis with optimal image quality and accuracy. In this guideline, to ensure proper opacification of the coronary arteries, the acquisition of CCTA images usually starts after reaching a predetermined threshold attenuation value in a predetermined (most cases related to the descending aorta) anatomical structure or after the enhancement is first displayed in the ascending aorta, by waiting for a certain delay time. Furthermore, the CT imaging operation can be triggered by analysis of the patient's ECG signal.

[0058] In step 202, the processor extracts data representing an axial trajectory extending along the coronary vasculature. For example, the axial trajectory may correspond to a centerline extending along the coronary vasculature. If the coronary vasculature represents a coronary artery, the axial trajectory may correspond to the centerline of the coronary artery, in which case the processor extracts the centerline of the coronary artery therefrom. The coronary artery centerline represents the center of the coronary artery lumen along the coronary artery section of interest. This can be a single coronary artery, a coronary artery branch, or a complete coronary artery tree. If the coronary artery section of interest includes one or more branches, the coronary artery centerline will include the branches but not their side branches.

[0059] When the bifurcation and / or coronary artery tree is analyzed, data representing multiple centerlines can be extracted in step 202. For the purposes of the present application, the extracted coronary artery centerlines do not need to accurately represent the center of the coronary artery lumen. They should not cross the coronary artery lumen, but a rough estimate of the coronary artery centerlines is sufficient. The extraction of the coronary artery centerlines can be performed manually or (semi)automatically. An example of a semi-automatic approach is described in Non-Patent Document 11. An example of an automatic extraction method for coronary artery centerlines is described in Non-Patent Document 12, which uses machine learning to automatically extract the coronary artery centerlines. This method places a single seed point on the artery of interest and then extracts the coronary artery centerline between the ostium and the most distal point visualized in the CCTA image dataset. In a preferred embodiment, a complete coronary artery centerline tree is automatically extracted, and each coronary artery segment is automatically labeled according to a model introduced, for example, by the American Heart Association (Non-Patent Document 13). This method is further described with reference to the flowchart of FIG. 28 and its high-level methodology shown in FIG. 29. The artery of interest can be identified by the user, or predefined, for example, as the left anterior descending artery (LAD), left circumflex artery (LCx), or right coronary artery (RCA), or automatically identified by the method described with reference to the flowchart of FIG. 28. FIG. 4 shows an example of a centerline tree (403) extracted from a CCTA dataset (401). The CCTA dataset is composed of a plurality of 2D images (401), and the resulting volume is generated (which is represented by the volume rendering 402 of this acquired volume). When there are multiple arteries, the method described in FIG. 2 or FIG. 17 or FIG. 23 can be automatically performed on multiple arteries of interest, such as the LAD, LCx, and RCA. These multiple arteries of interest can be predefined and automatically extracted by the method described in FIG. 28.

[0060] In step 203, using the data representing the axial trajectory (or centerline) that extends along the coronary artery extracted in step 202, a three-dimensional (3D) multi-planar reformation (MPR) image of the coronary artery of interest is created. FIGS. 5a-5d provide an illustration of the creation of the volumetric 3D MPR image. The image 501 in FIG. 5a shows a volume rendering of a CCTA data set (FIG. 2, 201) where the right coronary artery 502 is selected as an example for creating the 3D MPR image. There are linear MPR and curved MPR in the 3D MPR image. Similar to the case of curved MPR, in the case of linear MPR, the extracted axial (e.g., centerline) trajectory is used to create an isotropic 3D MPR image from the image data set 201. The resolution of the 3D MPR image is predefined and, for example, is 0.2 mm using trilinear interpolation. The 3D MPR image can also be created anisotropically (e.g., with an in-plane pixel size of 0.1 mm and a distance between consecutive centerline points of 0.25 mm) using, for example, trilinear interpolation.

[0061] For creating the 3D MPR image, it can involve defining a rectangular parallelepiped image 503 such that the centerline of the coronary artery is at the center of the rectangular parallelepiped image 504 and sampling the image data along the extracted axial trajectory 502 (e.g., the centerline of the coronary artery) (as a result, a linear MPR is created). The image 503 in FIG. 5b shows the resampled image of the rectangular parallelepiped (linear MPR), and for ease of interpretation, one "slice" is visualized within the resampled image of the rectangular parallelepiped. The image 505 in FIG. 5c shows one "slice" of the same resampled image, but the visualization plane shows the visualization of a coronary artery bifurcation (506) within the extracted right coronary artery rotated about the centerline 504.

[0062] Alternatively, creating 3DMPR images can be associated with sampling images along the curved course of the axial trajectory 502 (e.g., the coronary centerline) (as a result, a curved MPR is obtained). Images 508a and 508b in FIG. 5c show that the visualization plane is rotating within the rectangular resampled image (linear MPR). FIGS. 5d and 5e show two examples of curved 3DMPR images visualized as a single "slice". Here, the slice orientation refers to a surface that can be rotated along the curved coronary artery. Again, this is only for visualization purposes, and this application will use a complete 3D linear MPR image or a curved MPR image. The advantage of the curved MPR image is that the curvature or bend of the extracted centerline can be taken into account within the machine learning network architecture described within this application.

[0063] In an alternative embodiment, the MPR image created in step 203 can be represented by a two-dimensional (or "2D") MPR image of the coronary blood vessel.

[0064] In step 204, the MPR image of step 203 is supplied to a machine learning-based arterial characterization network. This network extracts data (signals) that characterize the features of the coronary blood vessel along the centerline of the coronary blood vessel, taking the MPR image as an input.

[0065] In an embodiment, the machine learning-based arterial characterization network of step 204 employs a convolutional neural network (CNN) architecture. A CNN architecture typically includes an input layer, a hidden layer, and an output layer. The hidden layer includes one or more layers that perform convolution. Typically, this includes a layer that performs the dot product of a convolution kernel and the input matrix of the layer. This product is typically a Frobenius inner product, and its activation function is generally a rectified linear unit (ReLU). As the convolution kernel slides along the input matrix of the layer, a feature map is generated by the convolution operation, which contributes to the input of the next layer. This is followed by other layers such as a pooling layer, a fully connected layer, and a normalization layer.

[0066] FIG. 6 shows an example of a CNN architecture for extracting arterial characteristics. For each point along the centerline of the coronary artery, the arterial characterization network predicts several arterial characteristics related to the severity of the arterial lesion based on the MPR image (601). In the arterial characterization network shown in FIG. 6, three characteristics are extracted by the CNN architecture. These characteristics or features include the cross-sectional lumen area, lumen attenuation (optional), and calcium area, all of which are specified along the axial trajectory of the coronary artery. An example of the CNN architecture is shown at 602 in FIG. 6. This analyzes a stack of a predefined amount of consecutive cross-sections (e.g., five cross-sections) slices within the MPR image and is composed of, for example, four alternating convolutional blocks and pooling operations. The convolutional block is composed of two convolutional layers (e.g., kernel size 3, 16 filters), followed by batch normalization and rectified linear unit (ReLU) activation functions in each layer. Finally, three separate output heads regress the values of the lumen area, average lumen attenuation, and calcium area of the central slice of the input stack, resulting in the output of arterial characteristics (in this example, lumen area, lumen attenuation, and calcium area) along the axial trajectory (e.g., centerline) of the coronary artery (FIG. 6, 603).

[0067] To train the arterial characterization network (204 in FIG. 2 or the CNN in FIG. 6), arterial reference values are used (FIG. 2, 208). The arterial reference values (208) are provided by a database that accommodates data of a plurality of patients. In each set within the database, for each patient, a) a contrast-enhanced CT image dataset (201 represents the reference image set during the training phase) and b) corresponding arterial characteristic reference values are accommodated. For the three characteristics described above, reference annotations of the lumen and coronary artery calcium of the coronary artery, from which the lumen area, calcified area, and lumen attenuation of the cross-section along the axial trajectory of the coronary artery are derived, are required. These reference annotations can be created by manual segmentation of the CCTA dataset used during training or by automatic segmentation of the lumen and calcium of the CCTA dataset used during training (with subsequent manual correction as needed). Automatic segmentation of the lumen and calcium can be performed, for example, by segmentation in the original CT image volume using the approaches described in Non-Patent Documents 14 and 15. After this, the automatic segmentation is transferred to the MPR of the artery, visually inspected, and corrected as needed. Since x-ray angiography has excellent image resolution compared to CCTA, in a preferred embodiment, the lumen segmentation on the MPR image is guided by the QCA3D analysis results extracted from the x-ray angiography image data of the same patient obtained within three months from the acquisition of the CCTA data of the patient in question. This QCA3D can be performed, for example, by the approach described in Non-Patent Document 16 using, for example, a CAAS Workstation (Pie Medical Imaging BV, Netherlands).

[0068] The arterial reference value (208) can be converted into the MPR image domain so that the arterial reference value is surely aligned with the spatial coordinates of the MPR image (203). If the arterial reference value (e.g., manual annotation of the severity of a plaque-type functional lesion such as FFR) is obtained by using the MPR image as a result of step 203, this step may be omitted. If the arterial reference value is obtained, for example, by annotation using a contrast CT dataset (step 201), this step converts the annotation into the MPR view. Such conversion is performed using the centerline extracted as a result of step 202. The conversion of the characteristics (205) of the coronary artery tree can be performed according to the above description.

[0069] Alignment is performed between the image dataset 201 and the invasively measured pullback FFR so that the coronary blood flow reserve ratio value (e.g., pullback FFR reference value) along the coronary artery measured in the catheterization laboratory is surely aligned with the spatial coordinates of the MPR image. To enable alignment of the pullback FFR measurement with the CT dataset, pullback movement information indicating the pullback speed while pulling back the FFR wire from the FFR wire start position (e.g., distal position of the coronary artery) to the FFR wire end position (e.g., proximal position of the coronary artery or the opening of the coronary artery) is obtained. The pullback movement information can be obtained by measuring the longitudinal movement of the FFR wire during pullback. The measurement value may be obtained by a movement measurement system or, for example, by using an electric pullback device that maintains a constant pullback speed. One or more processors of the system calculate the length of the pullback distance using the time required for the pullback of the FFR wire and the pullback speed. To align the pullback FFR arterial reference value with the MPR image, one or more processors convert the length of the pullback distance into the image dataset 703 used.

[0070] Figures 7a-7c show schematic diagrams of a method for aligning a pullback FFR reference value (e.g., pullback distance) with CCTA image data sets. Image 701 in Fig. 7a shows an x-ray coronary angiography image acquired in a catheterization laboratory. Image 702 in Fig. 7b shows a volume-rendered CCTA image of the same patient. Image 706 in Fig. 7c shows an x-ray fluoroscopy image without contrast agent.

[0071] The x-ray coronary angiography image 701 in Fig. 7a shows the right coronary artery where the FFR pressure wire is inserted until the pressure sensor can obtain the first / distal desired pressure measurement value at the desired distal position. Point 703 indicates the position of the pressure sensor on the FFR pressure wire in the x-ray angiography image at the distal position within the coronary artery before pullback. The position indicated by point 703 may also be referred to as the distal pressure sensor position. The distal position of the pressure sensor (and the entire FFR pressure wire) can be easily identified on the x-ray fluoroscopy image (without contrast agent, 706) by the radiopaque marker on the FFR wire 707 (shown in the image 706 of Fig. 7c), thereby enabling the identification of the position of the pressure sensor on the FFR wire.

[0072] Image 702 of FIG. 7b shows a volume-rendered CCTA image (belonging to the same patient from whom the pullback FFR reference value is obtained). In image 702, the right coronary artery 704 is identified, for example, as a result of step 202 of FIG. 2. Alignment of the pullback FFR reference value can be performed by identifying the position (705) of the FFR pressure wire before pullback within the CCTA data set (manually identifying, for example, that which is supported by anatomical landmarks such as the branch position), and then matching the lengths (the length of the 3D-extracted centerline and the length of the FFR pullback). Identifying the position of the FFR pressure wire before pullback within the CCTA data set can also be performed, for example, by registering the x-ray angiography image to the CCTA data set using the method of Non-Patent Document 17. Non-Patent Document 17 describes a method of registering a 2D x-ray angiography image to a 3D volume image data set (CCTA) using a Gaussian Mixture Model (GMM)-based point set registration technique. Identification of the position (707) of the pressure sensor on the FFR wire can be easily performed using the fluoroscopy image (706) by image processing techniques, so converting this position to the CCTA image data (705) can be easily done using the deformation field obtained from 2D / 3D registration, as described in Non-Patent Document 17.

[0073] In other embodiments, the machine learning-based arterial characterization network of step 204 can employ a variational autoencoder (VAE) architecture configured to extract features or characteristics of the coronary artery when a multi-planar reformation (MPR) image of the coronary artery is provided as input. Details of the variational autoencoder (VAE) architecture are described below with respect to step 1704 of FIG. 17 and FIGS. 19a and 19b.

[0074] In yet other embodiments, the machine learning-based arterial characterization network of step 204 can be configured to extract other data (signals) that characterize the features of the relevant blood vessels when the MPR image is provided as input. This can be achieved by including multiple arterial characteristics in the arterial characterization. For example, FIG. 8 shows an example of a CNN network architecture for extracting five arterial characteristics along the centerline of the relevant blood vessels based on the MPR image (801). The five arterial characteristics include the cross-sectional lumen area, lumen attenuation, calcium area, soft plaque area, and mixed plaque area. As shown in FIG. 8, during the training of the network, two additional reference features (soft plaque and mixed plaque) need to be extracted from the annotation compared to the network shown and described in FIG. 6. This annotation can be based on manual or (semi-)automatic segmentation of the CCTA image dataset. Automatic plaque segmentation can be based on, for example, the approach described in Non-Patent Document 18.

[0075] An example of this CNN architecture is shown at 802 in FIG. 8. This analyzes a stack of a predefined quantity of consecutive cross-sectional (e.g., five cross-sections) slices and is composed of, for example, four alternating convolutional blocks and pooling operations. The convolutional blocks are composed of two convolutional layers (e.g., kernel size 3, 16 filters), followed by batch normalization and ReLU activation functions in each layer. Finally, five individual output head regression values for the lumen area, average attenuation within the lumen, calcium area, soft plaque area, and mixed plaque area are provided, resulting in the obtained output arterial characteristics (in this example, lumen area, lumen attenuation, calcium area, soft plaque area, mixed plaque area) along the centerline of the relevant blood vessels (FIG. 8, 803).

[0076] The shape of blood vessels affects the characteristics of blood flow and the local appearance of blood vessels. Therefore, in step 205 of FIG. 2, other data (signals) characterizing the characteristics of the coronary blood vessels along the axial trajectory of the coronary blood vessels (obtained from step 202 of FIG. 2) can be extracted from the CCTA image data of step 201 of FIG. 2. For example, for each point along the trajectory in the direction of the axis (i.e., the centerline) of the coronary blood vessels, two additional characteristics can be extracted from the CCTA image data. The first characteristic indicates the presence of a branch (901 in FIG. 9) at the arterial centerline point. As described in step 202 of FIG. 2, the coronary artery centerline of the coronary blood vessels can be performed manually or (semi-)automatically. In a preferred embodiment, a complete coronary artery centerline tree is automatically extracted using the method shown in FIG. 29. Using this method, by identifying the centerline points along the centerline of the coronary artery where there are branch arteries, the presence and location of the branches along the centerline of the coronary artery can be automatically determined. Alternatively, the user can manually identify the branch artery (branch) within, for example, a CCTA slice (401 in FIG. 4) or a volume-rendered CCTA image (402 in FIG. 4). Yet another alternative method is to train a deep learning network of MPR images to automatically identify the branch artery (branch). The second additional characteristic indicates whether the centerline point belongs to the main branch (i.e., the left main branch (LM), LAD, LCX, RCA) or to the collateral branch (902 in FIG. 9). This can be automatically extracted, for example, using the automatic anatomical labeling method shown in FIG. 29 as described in step 202 of FIG. 2. Furthermore, the coronary blood vessels can be identified or selected by the user. If the coronary blood vessels contain collateral branches or branches, the user can identify the positions of the collateral branches or branches. These characteristics can be normalized to an average of 0 and a variance of 1 across the entire training dataset. Examples of additional characteristics characterizing the coronary blood vessels could also be the presence and / or type of stent (from previous coronary intervention procedures) at the centerline points of the blood vessels.

[0077] In step 206, the machine learning-based stenosis evaluation network is configured to evaluate the functional significance of stenosis using, as inputs, the feature data output by the first network (204 in FIG. 2 or the CNN in FIGS. 6 or 8) and other feature data extracted from the CCTA image data (205 in FIG. 2).

[0078] In an embodiment, the machine learning-based stenosis evaluation network can utilize the CNN network architecture described herein. FIG. 9 shows an example of a CNN network architecture for stenosis evaluation. The CNN architecture in FIG. 9 is generally composed of approximately three stages. In the first stage (903), first, the lumen area predicted by the characterization network (204 in FIG. 2) and its attenuation are pre-encoded, and then combined with the calcium area, and further combined with additional characteristics indicating the branches and whether the analysis is performed on the main artery or the collateral branches, which is the result of step 205 in FIG. 2. In the second stage (904), the combined encoding is sent to the encoder. In the encoder, first, the features are pooled, and then convolutional and transformer layers are applied. In the final classification, in the third phase (905), two separate output heads (a regression head and a classification head) are applied. In the regression head, the output of the second stage (904) is processed by two convolutional layers and a ReLU activation function. The resulting sequence is pooled along the dimension of the artery and subtracted from 1 to generate a single FFR value. In the classification head, the output of the second stage (904) is pooled, for example, to a fixed length of 2.5 mm. Then, two fully connected layers are used in combination with a sigmoid activation function to generate the output probability when there is a functionally significant stenosis in the artery.

[0079] The three stages of the machine learning-based stenosis evaluation network in FIG. 9 are described in detail below.

[0080] In the first stage (903), the network receives, as input, five arterial characteristics (lumen area, mean lumen attenuation (optional), calcium area, bifurcation, collateral). To focus on changes rather than the absolute values of the lumen area and its attenuation, the percentage difference with respect to the previous position at each location within the artery is calculated. Since the related features of the lumen area and its attenuation can be subtle and appear at various locations along the artery (i.e., due to stenosis, the attenuation distally from the appearance in the lumen area is expected to change), these two characteristics are initially encoded separately. This is done using two non - shared convolutional layers, with leaky rectified linear units (Leaky ReLU) applied between the layers. After this, the remaining characteristics are combined with the features encoded from the lumen area and its attenuation.

[0081] In the second stage (904), all of the information on the five extracted arterial characteristics is merged as follows by a common encoder composed of a convolutional layer and a transformer layer. To expand the receptive field and reduce the number of dimensions, for example, average pooling with a kernel size of 4 is applied, after which, for example, two convolutional layers with dilation 1 and 2 respectively are applied. After each convolutional layer, a leaky ReLU activation function, instance normalization, and dropout follow. After this, the encoding of the artery is combined with the original lumen area and its attenuation and sent to the transformer layer (Non - Patent Document 19). With the global receptive field, the transformer layer connects all arterial points to each other. This makes it possible to model the interaction between multiple lesions and the proximal and distal parts of the artery.

[0082] In the third stage (905), two separate output heads (a regression head and a classification head) are configured to perform separate tasks. The regression head performs regression of the FFR value, and the classification head performs classification of the presence of a functionally significant stenosis of the artery. Taking into account the additional property of the continuous flow resistance, the regression head is designed to predict the pressure drop along the artery. First, two convolutional layers (each followed by a leaky ReLU activation function, instance normalization, and dropout) are applied. After this, a ReLU activation function follows after a third convolutional layer with a single output filter map to enforce the positive value of the pressure drop. Finally, the predicted pressure drop is summed along the artery using a global pooling layer, and the resulting overall FFR drop is converted to the final FFR value (907) by subtracting it from 1. The output (906) of the classification head predicts the presence of a functionally significant stenosis (FFR ≤ 0.8). To explicitly relate the proximal and distal sections, first, for example, adaptive global pooling with five output features is applied, followed by, for example, two fully connected layers with leaky ReLU activation and dropout. Finally, a fully connected layer with a single output filter map and sigmoid activation generates the output probability for a functionally significant stenosis.

[0083] In an embodiment, for all convolutions of the entire stenosis evaluation network of FIG. 9, a kernel size of, for example, 3 can be employed in combination with zero-padding to prevent reduction of features. Further, for all convolutions and transformers, a relatively small number (e.g., 16) of filter maps are used to balance the required expressiveness and prevent overfitting. For the same purpose, all dropout probabilities can be set to, for example, 0.5.

[0084] To train the stenosis assessment network (206 in FIG. 2, or the CNN in FIG. 9), a reference standard can be used (FIG. 2, 209). The reference standard is a database that accommodates data of multiple patients. Each set in the database contains, for each patient, a) a contrast CT image dataset (201 represents the reference image set during the training phase), and b) corresponding CAD-related reference values representing hemodynamic indices for evaluating functionally significant coronary artery occlusions. For example, the CAD-related reference values may represent at least one of invasively measured coronary flow reserve ratio, coronary flow reserve, ratio without instantaneous wave, rest full cycle ratio, diastolic ratio, ratio without hyperemia, diastolic pressure ratio, Pd / Pa ratio at rest, hyperemic myocardial perfusion, index of microcirculation resistance, pressure drop along the coronary artery, fractional flow reserve along the coronary artery. It should be noted that the same contrast CT image dataset is required for the reference standard (209) and the arterial reference values (208).

[0085] During training, the regression head is monitored using, for example, the mean squared error with the CAD reference value FFR. Since the invasive reference FFR is often not measured at the most distal position, the contribution of the predicted pressure drop from the anatomical position distal to the measurement position is masked during training and testing. The measurement position is assumed to be, for example, 10 mm distal to the annotated lesion position according to the measurement protocol in clinical practice. The classification task is monitored using the binary cross-entropy loss function. The loss terms for the regression head and the classification head are equally weighted.

[0086] Finally, in step 207, one or more outputs are provided. In an embodiment, the output represents the probability that there is a functionally significant stenosis in the coronary artery. In yet another embodiment, the output represents the FFR as a value between 0.0 and 1.0. To combine the strength of the results of the classification head (906) and the results of the regression head (907), these outputs are merged into a single probability for the presence of a functionally significant stenosis in the coronary artery. The classification head directly predicts the probabilities of the positive and negative classes, while the regressed FFR values are distributed around a positive FFR threshold (≤0.8) and in the range [0.0, 1.0]. To enable these merges, the predicted FFR values are first converted to pseudo-probabilities by linearly scaling a symmetric window around the positive FFR threshold of 0.8 using the formula of Equation 1.

Number

[0087] To obtain the final prediction result of the output, the pseudo-probabilities can be averaged with the probabilities from the classification head.

[0088] Optionally, to enhance the robustness of the prediction result and to determine the uncertainty of the prediction result, the output of step 207 can be calculated as the average of multiple trained networks (both 204 and 205). This is done, for example, by performing randomized 10-fold cross-validation where 10 networks are trained on a random 90% subset and tested on the remaining 10%. During testing, we ensemble the networks by averaging the predicted probabilities and FFR values, as considered in Non-Patent Document 20, for example. For the prediction of uncertainty, the standard deviation of the probabilities and FFR values is calculated, as considered in Non-Patent Document 21, for example. The measure of uncertainty can be useful in clinical practice and can be used in a semi-automatic setting, for example, by introducing a hybrid approach. In particular, patients with arteries showing a high prediction uncertainty by this method may have invasive measurements referred.

[0089] Experimental setup

[0090] In an embodiment, the arterial characterization network (204 in FIG. 2 or the CNN in FIG. 6) was trained for 800 epochs using the mean absolute error as the loss function and a learning rate of 10 -5 , and the ADAMW (Non-Patent Document 22) optimizer with a batch size of 512. The loss term for lumen attenuation was scaled by a factor of 0.1. After training, this network was applied to each cross-section of the MPR to obtain the lumen area, its mean attenuation, and the area of calcium along the length of the artery.

[0091] In an embodiment, the stenosis assessment network (206 in FIG. 2 or the CNN in FIG. 9) was trained for 150 epochs using the ADAMW optimizer with a cyclic learning rate of a linear schedule. The cyclic learning rate varied between 5e-4 and 1e-5 over a cycle of 40 epochs. Since the lengths of the arteries vary and the stenosis assessment network can only process one artery at a time, the loss was accumulated over 8 training iterations before backpropagation of the error corresponding to an effective batch size of 8.

[0092] Expansion of the flowchart in FIG. 2.

[0093] This section describes some expansions of the flowchart of the machine learning-based method for determining the severity of functionally significant lesions in one or more coronary arteries, as described above with reference to FIG. 2.

[0094] Expansion 1 : FFR value for each centerline point.

[0095] The machine learning-based stenosis evaluation network described at 206 in FIG. 2 and further clarified in FIG. 9 can be configured to provide two outputs, namely, the regression FFR value for the entire coronary vessel and the binary classification of the presence of a functionally significant blood flow obstruction in the coronary vessel. This section describes an extension of the stenosis evaluation network that can be implemented by module 304 in FIG. 3. In particular, the architecture of the deep learning network can be adapted to provide the regression FFR value for the centerline points along the coronary vessel (as a result, the FFR value at the centerline points of the coronary vessel (FFR pullback graph) is obtained). FIG. 10 shows this extended architecture based on the stenosis evaluation network architecture shown in FIG. 9.

[0096] Regarding the description of the stenosis evaluation network (206 in FIG. 2 and further clarified in FIG. 9), the regression head is extended to include the FFR output for each centerline point. All other methodologies are the same as the description in the flowchart of FIG. 2. This extension focuses on the regression head, which is part of the stenosis evaluation network (FIG. 9). The regression head is designed to predict the pressure drop along the artery. First, two convolutional layers (each followed by a leaky ReLU activation function, instance normalization, and dropout) are applied. After this, a ReLU activation function follows a third convolutional layer with a single output filter map to enforce the positive nature of the pressure drop. This corresponds to predicting the FFR drop at each location or points along the centerline of the coronary artery (when given the multi-planar reformation (MPR) views shown in FIGS. 6, 601 and FIGS. 8, 801), and is performed by the "Accumulate" block shown in FIG. 10, 1001. This results in the FFR drop for each centerline point along the coronary artery, and this can be converted to the FFR value for each centerline point by subtracting it from 1.0 (FIG. 10, 1002), generating the resulting FFR pullback graph. The FFR values for each centerline point of the coronary artery can be presented so that the user can view them. For example, the FFR values for each centerline point can be plotted (drawn for visual inspection) along the coronary artery within the image of the coronary artery. In another example, the FFR values for each centerline point can be plotted (drawn for visual inspection) on the CCTA volume rendering of the coronary artery. In yet another example, the FFR values for each centerline point can be plotted (visually drawn) on a 3D model of the coronary artery. In yet another example, the FFR values for each centerline can be plotted (drawn for visual inspection) by color-coding a 3D representation of the coronary artery (either in the CCTA volume rendering view or 3D segmentation).

[0097] FIG. 11 shows an exemplary graphical user interface (display screen) for visually communicating such information to a user. 1102 in FIG. 11 shows a volume rendering image of CCTA data in which three major coronary arteries, the right coronary artery (RCA), left anterior descending coronary artery (LAD), and left circumflex coronary artery (LCX), are emphasized. The RCA is the coronary artery of interest, and this results in the multi-planar reconstruction (MPR) shown in part B of 1101 in FIG. 11. In this figure, the left side (1103) corresponds to the position of the opening (proximal) of the RCA, and the right side (1104) corresponds to the distal position of the RCA. Part C of 1101 shows a graph of the lumen area or lumen diameter along the RCA. The x-axis of this graph corresponds to the x-axis of the MPR view in part B of 1101 and the centerline of the RCA. Finally, part D of 1101 shows the FFR values along the centerline points of the RCA. The x-axis of this graph is the same as the x-axis of the MPR view and the diameter / area graph. Further, in 1102, the corresponding segments of the volume-rendered RCA are color-coded using the FFR values along the centerline of the volume-rendered RCA. It should be noted that the vertical markers in parts C and D of 1101 correspond to the minimum area / diameter (thick line marker) and occlusion range (dashed line marker) obtained from general quantitative coronary analysis techniques as described in Non-Patent Document 16.

[0098] To train the stenosis evaluation network shown in FIG. 10 to output FFR values for each centerline point, a reference FFR value (209) can be measured along the coronary artery of interest (which will result in a coronary flow reserve ratio value invasively measured at each position along the coronary artery centerline). This can be obtained by performing a manual or motorized pullback during the measurement of the coronary flow reserve ratio. In a catheterization laboratory, an interventional cardiologist or physician places an FFR wire at a distal position within the coronary artery of interest. During an automatic or manual pullback, the FFR value is continuously measured until the FFR wire reaches the coronary artery opening (Non-Patent Document 23).

[0099] As an option, alignment can be performed to confirm that the values of the coronary blood flow reserve ratio along the coronary artery measured in the catheterization laboratory (e.g., pullback FFR reference values) are aligned with the spatial coordinates of the MPR image. This can be performed, for example, by the method described in step 208 of FIG. 2.

[0100] As an option, when it is possible to use x-ray angiography image data for training, the reference FFR per centerline point value can be calculated based on 3D coronary artery reconstruction using x-ray angiography, as taught in, for example, Patent Document 5. Patent Document 5 describes a method for calculating a vFFR pullback along the coronary artery of interest based on 3D coronary artery reconstruction. Since the spatial resolution of x-ray angiography is high, the accuracy of vFFR is quite high, as described in Non-Patent Document 24, where the CAAS Workstation 8.0 (Pie Medical Imaging, Netherlands) was used to obtain the vFFR value and the pullback vFFR value. In an alternative embodiment, since a reference standard (209 in FIG. 2 or 1609 in FIG. 16) as an FFR reference value is obtained by calculation of the FFR pullback or calculation of the distal FFR value, it is not necessary to invasively measure hemodynamic parameters such as FFR. This can be performed by using the patient's x-ray angiography image data and calculating the (pullback) FFR value using, for example, the vFFR (vessel - FFR) workflow in the CAAS Workstation.

[0101] The vFFR method of the CAAS Workstation generates 3D coronary artery reconstructions using at least two angiographic x-ray projections that are at least 30 degrees apart. vFFR is calculated instantaneously by using a proprietary algorithm that incorporates the morphology of the 3D coronary artery reconstruction and the patient-specific aortic pressure measured routinely. Figure 12 shows an example of obtaining the calculated FFR pullback of the coronary circumflex artery using the CAAS Workstation. 1201 shows the segmentation of the coronary circumflex artery in each 2D x-ray angiographic image (which results in generating a 3D reconstruction (1202) of the coronary artery). Graph 1203 shows the vFFR values calculated along the length of the 3D reconstructed coronary artery. Since the same approach can also be performed on the CTA dataset, the corresponding x-ray angiographic image data is not required for each patient.

[0102] Expansion 2 : FFR values and additional arterial characteristics for each centerline point.

[0103] In this section, Expansion 1It is extended by another extension of the workflow method described in FIG. 2. This extension can be implemented in module 304 of FIG. 3. In particular, the architecture of the stenosis assessment network (the stenosis assessment network of FIG. 9) employs five different arterial characteristics as inputs. Three of the five arterial characteristics (lumen area, lumen attenuation, calcium area) are obtained from the MPR images of the coronary vessels (step 203), and two of the five arterial characteristics (collaterals, bifurcations) are obtained from the extracted coronary centerline tree (step 205). The method described in FIG. 2 is not limited to the described vascular characteristics. And this can be easily extended not only to additional vascular characteristics obtained from any of the analysis of MPR images, the analysis of coronary vessel image data, or the analysis of the coronary centerline tree, but also to other characteristics that describe the coronary vessels. For example, data representing the soft plaque area and the mixed plaque area of the coronary vessels, which can be derived using the method described in Patent Document 6 (Method and System for Evaluating Vascular Occlusion Based on Machine Learning), can be used as data (signals) that characterize the characteristics of the coronary vessels for the input to the stenosis assessment network. In summary, Patent Document 6 first extracted the centerlines of the coronary arteries in the CCTA images. These were used for the reconstruction of the extended multi-sectional reconstructed images of the coronary arteries. To perform automatic analysis, a multi-task recursive convolutional neural network was applied to the coronary artery multi-sectional reconstructed images, and two simultaneous multi-class classification tasks were performed. In the first task, the network detected and characterized the type of coronary plaque (no plaque, non-calcified, mixed, calcified). In the second task, the network detected and determined the anatomical significance of coronary stenosis (no stenosis, not significant, i.e., less than 50% lumen stenosis, significant, i.e., 50% or more lumen stenosis) and / or the severity of functional coronary lesions.

[0104] FIG. 13 shows an example in which data representing the soft plaque area and the mixed plaque area of the coronary artery is integrated into the architecture of the deep learning network of FIG. 10. In FIG. 13, the soft plaque area (1301) and the mixed plaque area (1302) are added as characteristics of the coronary artery. Optionally, the data representing the soft plaque area and the mixed plaque area of the coronary artery can be pre-encoded before passing through the encoder. Further, the data describing one or more characteristics of the coronary artery can be a binary signal (1 if a soft plaque or a mixed plaque is present and 0 if not) along the MPR image of the coronary artery.

[0105] In addition or alternatively, other data (signals) describing the characteristics of the coronary artery can be integrated into the architecture of the deep learning network of FIG. 2. For example, data (signals) related to the image quality of the MPR view of the coronary artery can be integrated into the architecture of the deep learning network. For example, such data (signals) can indicate the presence of a step-like artifact and / or represent the noise level within the MPR view. In another example, data (signals) related to the relative lumen attenuation of the coronary artery (when compared to another major coronary artery) can be integrated into the architecture of the deep learning network. For example, when the coronary artery represents the RCA, the data (signals) can represent the relative attenuation of the RCA compared to the LAD and / or the relative attenuation of the RCA compared to the LCX. This signal would be related to the blood flow difference between the main coronary arteries, which is expected to be present when there is a functionally significant reduction in blood flow due to epicardial occlusion and / or (localized) microvascular disease.

[0106] Expansion 3 : Include myocardial characteristics

[0107] Patent Document 6 recognized that the acquisition of CCTA images usually starts when a predetermined threshold attenuation value reaches a predetermined anatomical structure (most often related to the descending aorta) or after waiting for a certain delay time after the enhancement first appears in the ascending aorta. Thereby, when the injected contrast agent comes to exist in the coronary arteries, this will be delivered to even smaller generations of the coronary arteries and will flow from there into the microvascular system of the coronary arteries (this will lead to the enhancement of the myocardium). Due to the above-described acquisition method of CCTA, functionally significant coronary artery stenosis causes ischemia of the ventricular myocardium, so a difference occurs in the myocardial texture characteristics between the normal part and the ischemic part of the myocardium at the time of CCTA image acquisition. Patent Document 6 describes a method for detecting the presence of functionally significant stenosis in one or more coronary arteries based on machine learning using only the characteristics of the myocardium. In summary, in Patent Document 6, first, the myocardium of the CCTA image is segmented. Next, from the segmented myocardium, encodings are extracted in an unsupervised manner using a convolutional autoencoder, and these are used to calculate features. The convolutional autoencoder contains two parts, an encoder and a decoder. The encoder compresses the data into a low-dimensional representation by convolutional layers and max-pooling layers. The decoder unfolds the compressed form and reconstructs the input data by transposed convolutional layers and upsampling layers. To represent the entire myocardium, the statistics of the encodings of all voxels within the myocardium are used as features. Finally, based on the extracted features, the patient is classified using a support vector machine into patients with and without functionally significant coronary artery stenosis.

Number

[0108] This section describes another extension of the method described in the flowchart of FIG. 2 or FIG. 17. This can be implemented by module 304 of FIG. 3. In particular, the architecture of the stenosis assessment network described in block 206 of FIG. 2 or block 1706 of FIG. 17 can be adapted to use data characterizing the myocardium of the heart.

[0109] FIG. 14 provides an example of integrating characteristics of the myocardium of the heart obtained from CCTA image data using, for example, the method described in Patent Document 6. Data representing such characteristics of the myocardium are used as input to the encoder of the stenosis assessment network (206 or 1706). For example, the feature vector (1401) obtained from the myocardial analysis described in Patent Document 6 can be provided as input to the encoder (904 of FIG. 9 or 2003 of FIG. 20). This can be done by treating the feature vector as an additional input to the encoder (1402), or by combining the feature vector with the result of the encoder immediately before one of the fully connected layers of the classification head (1403), or by combining the myocardial feature vector obtained from the convolutional encoder with other input data (1403).

[0110] The above Expansion 2 It should be noted that the arterial characteristics described can also be used as "additional information" (block 155 of FIG. 15 of Patent Document 6) within the method described in Patent Document 6.

[0111] Alternatively, instead of integrating the feature vectors from myocardial analysis, the FFR classification results described in Patent Document 6 can also be integrated into the stenosis evaluation network of FIG. 14. In this case, the FFR myocardial classification (for example, the result of block 158 in FIG. 15 of Patent Document 6) can be directly coupled to the "classification head" (1404) of FIG. 14. The myocardial feature vectors described in Patent Document 6 are based on the entire myocardium (e.g., block 156 in FIG. 15 of Patent Document 6). The method described in Patent Document 6 can be adjusted to focus on the regions of the myocardium covered by the coronary vessels. Two possible approaches are described below.

[0112] Approach 1 :

[0113] The myocardial region covered by the coronary vessels can be defined, for example, by applying the Voronoi algorithm (Non-Patent Document 25) to the extracted axial trajectory of the coronary vessels using the method of the flowchart in FIG. 28. FIG. 15 shows an example of dividing the myocardium of the heart into regions covered by the coronary arteries. In this example, the coronary vessels are identified by the black arteries (the second LAD diagonal line identified by the black arrow, 1501), and the myocardial region covering this artery is identified by the pink region (1502 identified by the gray arrow). If the coronary vessels cover a plurality of myocardial regions identified by the Voronoi algorithm, these plurality of regions can be merged into one region.

[0114] In this approach, the calculation of the feature vectors described in Patent Document 6 is limited to a defined area, and the resulting myocardial feature vectors are integrated into the stenosis evaluation network (206), as shown in FIG. 16, for example. In FIG. 16, the myocardial feature vectors are regarded as additional input signals and pre-encoded using a convolutional network (1606), and then this additional encoding is sent to the encoder. To calculate the myocardial features, for each centerline point (1605 or 2201 in FIG. 22) along the coronary artery, the features are aggregated from the corresponding myocardial area (1602 or 2202 in FIG. 22). This is done, for example, by interpreting the corresponding myocardial area as a single cluster and using the standard deviation over the features from all voxels. Alternatively, this is done by interpreting the corresponding myocardial area as multiple clusters, generating a feature vector for each cluster using the standard deviation, and then creating one feature vector for the corresponding myocardial area using the maximum value over all clusters. Another method is to define the corresponding myocardial area by the part of the myocardium that is perfused by all downstream segments at a particular centerline point and repeat this for all centerline points within the coronary artery.

[0115] Approach 2 :

[0116] In the case of this second approach, the calculation of the feature vectors described in Patent Document 6 can be performed at each centerline position along the coronary artery or within a small area around it, and the resulting feature vectors can be used as additional arterial characteristics input to the encoder of the stenosis evaluation network. Alternatively, the feature vector calculation described with reference to FIG. 15 of Patent Document 4 can be performed, and the resulting feature vectors can also be used as additional arterial characteristics (1605 or 2201 in FIG. 22) input to the encoder of the stenosis evaluation network as shown in FIG. 16. Optionally, such new arterial characteristics can also be applied to the pre-encoder (1606 or 2202 in FIG. 22).

[0117] Myocardial ischemia occurs when blood flow to the heart muscle (myocardium) is impeded by partial or complete occlusion of the coronary arteries due to plaque accumulation (atherosclerosis). As a result, patients typically experience chest pain (angina). Up to half of the patients undergoing selective coronary angiography for the investigation of chest pain do not show evidence of obstructive coronary artery disease. These patients are often discharged with a diagnosis of non-cardiac chest pain, but there is a significant possibility that the cause of the patient's symptoms is ischemic. This type of ischemic chest pain in the absence of obstructive coronary artery disease is called ischemia with non-obstructive coronary arteries (INOCA). INOCA is related to an imbalance between myocardial oxygen supply and demand caused by microvascular dysfunction. Microvascular dysfunction is related to the dysfunction of small blood vessels that supply blood to the myocardium and is particularly common in middle-aged women. INOCA can also be caused by vasospastic diseases caused by coronary artery spasm.

[0118] The advantage of the method described in Patent Document 6 is that it can distinguish between both obstructive coronary ischemia and non-obstructive coronary ischemia. However, from the perspective of patient treatment, since the treatment strategies for obstructive coronary ischemia and non-obstructive coronary ischemia are different, it is necessary to identify the differences.

[0119] By integrating the method described in Patent Document 6, it becomes possible to integrate the identification of microvascular dysfunction into the stenosis evaluation network.

[0120] Since INOCA is microvascular dysfunction without epicardial coronary artery occlusion, this can be identified by examining the relevant blood vessels when the output of a deep learning network (e.g., the network provided in Figures 9, 10, 13, 14, 16, 20, 21, or 22) functionally significantly identifies a blood flow occlusion. This can be done in the following way. ● Examination of arterial characteristics obtained from the "artery characterization" network (Figure 6 or Figure 8 or Figures 19a and 19b). For example, execution on the lumen area graph of QCA analysis. For example, if the severity of the occlusion exceeds 50%, this is classified as obstructive coronary artery disease. ● Execution of QCA analysis for the segmentation of the coronary artery tree, for example, by the method described in Non-Patent Document 26. Here, the QCA analysis is executed, for example, by the method described in Patent Document 7. For example, if the severity of the occlusion of the coronary artery tree is identified as exceeding 50%, this is classified as obstructive coronary artery disease. ● The above QCA analysis and Expansion 2 combination with the amount of coronary artery plaque obtained by or by any of the methods described in Patent Document 4 or Non-Patent Document 14 or Non-Patent Document 27. ● Addition of a classifier to the network architectures of Figures 9, 10, 13, 14, 16, 20, 21 or 22. This can include adding another "classification INOCA head" similar to the "classification head". This head is trained to output values characterizing microvascular dysfunction, such as, for example, the Index of Microcirculatory Resistance (IMR) and / or coronary blood flow reserve capacity.

[0121] Optionally, the method described in Patent Document 6 (and the method integrated into the deep learning network described herein) can be refined by including a CT calcium scan. In contrast to a CCTA scan, a CT calcium scan is acquired without injecting a contrast agent. Incorporating a CT calcium scan provides information on the myocardium in the absence of contrast fluid (resulting in "baseline myocardium"). By incorporating a CT calcium scan, the machine learning network can integrate the intensity of the myocardium without contrast enhancement, thereby refining the detection of subtle contrast changes between healthy and ischemic myocardial regions. After the CT calcium scan is registered with the CCTA scan, both image datasets can be used in the method described in Patent Document 6.

[0122] Figure 17 shows another embodiment of the present application. The steps shown here can clearly be performed in any logical order and some can be omitted. Figure 17 shows a flowchart of a machine learning-based method for determining the severity of functionally significant lesions along the axial trajectory of the coronary vessels. The machine learning-based method is shown in Figures 2 and Expansion 2Although it is similar to the method described with reference to "FFR values and additional arterial characteristics for each centerline point", there are some fundamental differences. To enable a fast and quantitative evaluation of the CAD distribution, a deep learning-based method is used for predicting FFR pullbacks when MPR images of the coronary arteries are input. This method consists of two stages, and an overview is illustrated in FIG. 18. First, the axial trajectory (e.g., centerline) of the coronary artery is extracted (1801) from the input CCTA image, and an MPR image is generated from the axial trajectory of the coronary artery and the CCTA image data of the coronary artery. In the first stage of the deep learning-based method, the coronary artery is characterized (1802) along the length of the MPR image. In an embodiment, this first stage can combine unsupervised learning and supervised learning. Unsupervised learning provides the extraction of features that are not manually created from the MPR image. Supervised learning explicitly incorporates clinical knowledge into the machine learning model. Subsequently, the features characterizing the coronary artery derived from both the supervised learning method and the unsupervised learning method are sent to a deep learning network (1803), and the FFR drop along the artery is predicted in the second stage (1804). In order to be able to distinguish between focal CAD and chronic CAD, it is important that the prediction of the FFR drop along the axial trajectory of the coronary artery is accurate. This is achieved by using the FFR pullback reference (1709, FIG. 17) to perform supervised learning of the FFR pullback prediction. For this purpose, a new loss function inspired by the Earth Mover’s Distance (EMD) is used. Since it is difficult to register the reference FFR signal to the MPR image, it is not possible to eliminate the mismatch. Therefore, directly using a loss function such as the Mean Absolute Error (MAE) that evaluates all vascular points individually may generate an ambiguous loss signal when there is little overlap between the reference FFR drop and the actual FFR drop of the input artery. Instead, the proposed loss function continuously increases together with the distance between the predicted FFR drop and the reference FFR drop.The EMD loss focuses on the FFR drop around the assumed lesion location, but due to the uncertainty of the correct lesion location, the steepness of the FFR drop tends to be underestimated. To distinguish between focal CAD and chronic CAD, since it is necessary to correctly predict the steepness of the FFR pullback, an additional loss function that penalizes the histogram of the FFR drop has been designed. A deep learning-based method for predicting the FFR pullback will be described in detail with reference to FIG. 17.

[0123] In the first step 1701, a CCTA image dataset of the coronary artery of interest is obtained. Such a CCTA image dataset represents a volumetric CCTA image dataset (e.g., a single contrast-enhanced CCTA dataset) and is identical to the description of step 201 in FIG. 2.

[0124] In step 1702, an axial trajectory extending along the coronary artery of interest is extracted. This step is identical to the description of step 202 in FIG. 2.

[0125] In step 1703, three-dimensional (3D) multi-planar reformation (MPR) images of the coronary artery of interest are created. This step is identical to the description of step 203 in FIG. 2.

[0126] In steps 1704 and 1705, the first stage of the deep learning-based method is adopted. In this first stage, the features of the coronary artery of interest can be extracted by a combination of unsupervised learning and supervised learning.

[0127] In an embodiment, this first stage adopts an arterial feature characterization network (e.g., 1704 in FIG. 17 and 1802 in FIG. 18) that characterizes the features of the coronary artery of interest using unsupervised machine learning with the MPR image of step 1703 as the input.

[0128] In an embodiment, a variational autoencoder (VAE) that can be configured to extract features not manually created from MPR images can be adopted for an artery characterization network (e.g., 1704 in FIG. 17 and 1802 in FIG. 18). As described in Non-Patent Document 28, the variational autoencoder (VAE) is a generative model that approximates a data generation distribution. Through approximation and compression, the resulting model captures the underlying data manifold (a constrained, smooth, continuous, low-dimensional latent (feature) space where the data is distributed) (Non-Patent Document 29). Since the VAE enforces latent features with independent normal distributions, it enhances the interpretability and density of the latent space compared to conventional convolutional autoencoders. These advantageous techniques of the latent space are used within the machine learning-based Artery Characterization Network described in this step.

[0129] A typical VAE includes two main parts: an encoder and a decoder. The encoder compresses (encodes) the data into a low-dimensional latent space through convolutional operations and downsampling (max pooling), and then decompresses (decodes) this compressed form to reconstruct the input data through transposed convolutional operations and upsampling (unpooling). When the VAE is trained to minimize the distance loss between the encoder input and the decoder output, the abstract encoding generated from the input will surely contain sufficient information to reconstruct with low error. Once the VAE is trained, the decoder is discarded, and the encoder is used to generate encodings of unknown data.

[0130] Figures 19a and 19b show an example of the architecture of a VAE network that characterizes the features of the coronary vessels when an MPR image (from step 1703) is provided as input. The VAE network includes a convolutional encoder (1902). This encoder is configured to calculate several features from a stack of a specified quantity of consecutive cross-sectional slices of the input MPR image (1901) (e.g., 13 cross-sectional slices). The convolutional encoder (1902) can accommodate convolutional blocks that include, for example, two convolutional layers, batch normalization, and ReLU activation. The second convolutional layer of each block can have a stride of, for example, 2 to enable downsampling of the resolution. To increase the amount of context information, the outputs of the network after the second and fourth convolutional blocks can be combined ("fused") with the outputs of the same network applied to, for example, the sixth and third adjacent MPR slices respectively (thereby expanding the receptive field in the z direction). Further, for example, before the first convolutional block, after the third convolutional block, and after the last convolutional block, branch information of the coronary vessels (1903) can be inserted into the convolutional encoder. In an embodiment, the branch information can be a branch feature map indicating whether the MPR slice is located at a branch, and the branch feature map can be injected into each convolutional block by combination. The branch information can be generated or derived and provided by block 1705 of FIG. 17 using the method described with respect to step 204 of FIG. 2.

Number

[0131] To configure the encoder (1902) to extract relevant information from any input MPR image (1901) and obtain an encoding representing the distribution of latent features within the input MPR image, the encoder (1902) and the main decoder (1904) are based on the encoding output by the encoder (1902)

Number

[0132] To enhance the machine learning network and extract features characterizing the shape of arteries and plaques, the auxiliary decoders (1905a, 1905b) are configured to process the encoding (or a part thereof) output by the encoder (1902) to predict segmentation masks of lumen characteristics and calcified and non-calcified plaque characteristics. The auxiliary decoder 1905a can concretize the feature data characterizing the lumen area of the relevant blood vessel by processing the encoding output by the encoder (1902) through regression and outputting it via a linear layer across the axial trajectory (1906) of the relevant blood vessel. The auxiliary decoder 1905b can have an architecture similar to that of the main decoder (1904) with a different last layer. The last layer has four channels and uses softmax activation to output feature data characterizing the lumen attenuation, calcified plaque, and non-calcified plaque of the relevant blood vessel across the axial trajectory of the relevant blood vessel. Optionally, the auxiliary decoders (1905a, 1905b) can incorporate a plurality of linear layers configured to extract feature data characterizing the shape of arteries and plaques from the encoding

Number

Number

[0133] Furthermore, the encoder (1902) can generate an encoding output by the encoder (1902), for example, consisting of 32 predetermined amounts of predicted mean and standard deviation (1907). The encoding output by the predicted mean and standard deviation (1907)

Number

Number

Number

Number

[0134] The VAEs of FIGS. 19a and 19b provide a machine learning-based arterial characterization network (1704 of FIG. 17) that combines the supervised and unsupervised features extracted from the MPR images and the coronary artery tree. FIG. 19b illustrates in detail the unsupervised and supervised parts of the VAE. Both FIGS. 19a and 19b show the VAE architecture of the machine learning-based arterial characterization network. A stack of a predetermined number of slices around the central MPR slice at a specific position along the centerline of the coronary vessel is encoded using a convolutional encoder (1902). The encoder (1902) is composed of, for example, five convolutional blocks, where each block is composed of two convolutions. The branch information (1903) can be inserted into the convolutional blocks of the encoder (1902) as described above. At the output of the encoder (1902), the latent vector encoding

Number

Number

Number

[0135] To train the monitoring part of the VAE (1951 in Fig. 19b), a reference value (1708 in Fig. 17) can be used. The reference value can be obtained from a database that accommodates data of a plurality of patients. Each set in the database accommodates, for each patient, a) a contrast CT image data set (1701 represents the reference image set during the training phase) and b) corresponding arterial characteristic reference values for this. The arterial characteristics include reference annotations of the lumen of the coronary artery and plaques (calcified, non-calcified). These reference annotations can be created by manual segmentation of the CCTA data set used during training or by automatic segmentation of the lumen and plaque components of the CCTA data set used during training, and are manually corrected as necessary. The automatic segmentation of the lumen and plaque components can be performed, for example, by segmentation within the original CT image volume using the approaches described in Non-Patent Documents 14 and 15. After this, the automatic segmentation is transferred to the MPR image of the artery, visually inspected, and corrected as necessary. Since X-ray angiography has better image resolution than CCTA, in a preferred embodiment, the lumen segmentation on the MPR image is guided by the QCA3D analysis results extracted from the X-ray angiography image data of the same patient obtained within 3 months from the acquisition of the CCTA data of the patient in question. QCA3D can be performed, for example, by a CAAS Workstation (Pie Medical Imaging BV, Netherlands) using the approach described in Non-Patent Document 16.

[0136] The reference value can be converted into the MPR image domain so that the reference value 1708 aligns reliably with the spatial coordinates of the MPR image (1703). If the reference value (e.g., lumen and plaque type annotations) is obtained by using the MPR image resulting from the step 1703, this step may be skipped. If the reference value is obtained by annotation using, for example, contrast-enhanced CT dataset (step 1701), this step converts the annotation into the MPR view. Such conversion is performed using the vascular trajectory extracted as a result of step 1702.

[0137] In an embodiment, during the training of the machine learning-based arterial characterization network (1704 in FIG. 17 and 1802 in FIG. 18), three individual loss functions can be used. The mean absolute error (MAE) loss function can be used to minimize the loss of the MPR reconstruction loss (1904). Similarly, the MAE loss function can be used to monitor the regression of the lumen feature (1906). For the segmentation loss (1905), the binary cross-entropy loss function can be used.

[0138] In an embodiment, the latent space encoding of the VAE

Number

[0139] In VAE, the output of the encoder (1907) predicts the mean and standard deviation of each feature, and then a random sample is drawn from a normal distribution parameterized by this mean and standard deviation to generate a latent space encoding.

Number

[0140] As an option, the VAE auxiliary output decoder (1905b) can be left connected during the processing of non-visible data, and the predicted lumen, calcified plaque, and non-calcified locations can be used to calculate geometric parameters from the relevant blood vessels using quantitative coronary analysis (QCA). First, a 3D model is created from the segmentation regions (results of the VAE auxiliary output decoder (1905)), such as the blood vessel lumen, blood vessel plaque (calcified and non-calcified), in either the spatial coordinate system of the MPR image (1703) or the spatial coordinate system of the CT image (1701). Next, anatomical results are calculated from this 3D model using, for example, the method described in Non-Patent Document 16. Examples of such quantitative anatomical results are length, equivalent diameter along the axial trajectory of the relevant blood vessel, cross-sectional area along the axial trajectory of the relevant blood vessel, occlusion length, minimum equivalent diameter, minimum lumen area, diameter stenosis rate, area stenosis rate, reference diameter / area, blood vessel volume, plaque (calcified, non-calcified) volume, plaque burden (plaque volume / blood vessel volume). A healthy reference diameter or area graph representing the diameter / area when the blood vessel is healthy is calculated, for example, by fitting a line through the diameter or area values along the axial trajectory of the relevant blood vessel, as described in Non-Patent Document 30 (the diameter or area values within the lesion range are excluded during fitting).

[0141] Alternatively, blood vessel characteristics such as lumen attenuation, lumen area, calcium area, soft plaque area, and mixed plaque area along the axial trajectory of the relevant blood vessel can be extracted as described by the method of step 204 in FIG. 2 and further described with reference to FIG. 8.

[0142] In an embodiment, this first stage (1704) of the deep learning-based method can also use supervised machine learning to characterize the features of the coronary vasculature, taking the MPR image of step 1703 as input. Additional features may be defined to account for the influence of the arterial shape and local appearance on blood flow. Specifically, at step 1705 of FIG. 17, other data (signals) characterizing the features of the coronary vasculature along the axial trajectory (e.g., centerline) of the coronary vasculature (obtained from step 1702 of FIG. 17) can be extracted from the CCTA image dataset of step 1701. For example, for each point along the axial trajectory of the coronary vasculature, two or more additional features can be extracted. One of the additional features can indicate the presence of a branch at the arterial centerline point. Another additional feature can indicate whether the centerline point belongs to the main branch (i.e., left main (LM), LAD, LCX, RCA) or a collateral branch. Other examples of additional features characterizing the coronary vasculature could be the presence of a stent and / or the type of stent (from a previous coronary intervention procedure) at the vascular centerline point.

[0143] The second stage of the deep learning-based method (steps 1706 of FIG. 17 and 1803 of FIG. 18) employs a machine learning-based FFR pullback network configured to characterize the FFR pullback along the axial trajectory of the coronary vasculature based on the features of the coronary vasculature output from the arterial characterization network (1704) and the characteristics of the coronary tree (1705).

[0144] In an embodiment, the machine learning-based FFR pullback network can utilize the CNN network architecture described herein. FIG. 20 shows an example of a CNN network architecture for characterizing an FFR pullback along the axial trajectory of a coronary vessel. The input signal is represented by 2001. To remove the trend of the lumen area signal, the percentage difference is calculated for each position along the axial trajectory of the coronary vessel with respect to the value of the previous position. The calculated percentage difference signal of the lumen area and the VAE encoding are pre-encoded separately (2002) to extract features independent of the local environment. A plurality (e.g., four) of convolutional layers that gradually expand to enlarge the receptive field are applied to the lumen area signal to enclose the entire coronary vessel. The VAE encoding is preferably pre-encoded with a small number (e.g., two) of convolutional layers to prevent overfitting. After that, we combine the resulting features with the remaining characteristics (2003), and input the resulting encoding into a common convolutional path for regression of the FFR pullback with convolutional layers, average pooling, and ReLU activation functions (2004). The FFR drop regression network (2004) gets hints from additional properties of consecutive blood flow resistances. The FFR drop regression network first predicts the FFR pullback by predicting the FFR drop for each point of the coronary artery (2005). Thus, the prediction target at each point is independent of the previous output. After that, the FFR pullback (2006) is calculated by summing the cumulative predicted FFR drops. To obtain the arterial-level FFR prediction (2007) from the predicted FFR drop, the minimum value of the FFR pullback that is the same as the most distal value is adopted. The FFR pullback prediction network uses, for example, a kernel size of 3 for all convolutions including zero-padding to maintain the feature size. Further, we use, for example, 16 filter maps to balance expressiveness and overfitting. We use average pooling to account for potential misalignment with the reference FFR drop for each point.Furthermore, to prevent overfitting, the dropout probability is set to, for example, 0.5, and instance normalization is used throughout the FFR regression network.

[0145] To train the machine learning-based FFR pullback network (1706 in FIG. 17 and 1803 in FIG. 18), a reference standard is used (1709 in FIG. 17). The reference standard can be provided from a database containing data of multiple patients. Each set in the database contains, for each patient, a) a contrast-enhanced CT image dataset (1701 represents the reference image set during the training phase), and b) CAD-related reference values representing hemodynamic indices for evaluating functionally significant coronary artery occlusions corresponding thereto. For example, the CAD-related reference values can represent at least one of invasively measured coronary flow reserve ratio, coronary flow reserve, instantaneous wave-free ratio, rest-to-whole cycle ratio, diastolic hyperemic ratio, diastolic pressure ratio, rest Pd / Pa ratio, hyperemic myocardial perfusion, an index of microcirculatory resistance, pressure drop along the coronary artery, and coronary flow reserve ratio along the coronary artery. It should be noted that the reference standard (1709) and the arterial reference values (1708) require the same (reference) contrast-enhanced CT image dataset.

[0146] Since the focus is on predicting FFR along the axial trajectory of the coronary artery of interest, the reference standard needs to represent the FFR along the axial trajectory of the coronary artery of interest. This can be obtained from manual or motorized invasive FFR pullback. In the catheterization laboratory, an interventional cardiologist or physician places the FFR wire at a distal position within the coronary artery of interest. During automatic or manual pullback, the FFR value is continuously measured until the FFR wire reaches the coronary artery ostium (Non-Patent Document 31). Optionally, if x-ray angiography image data can be used for training, for example, as taught in Patent Document 5 and herein Expansion 1As described in detail in the description, the reference FFR around the center line point value can be calculated based on 3D coronary artery reconstruction using x-ray angiography. Patent Document 5 describes a method for calculating a vFFR pullback along a coronary artery of interest based on three-dimensional coronary artery reconstruction. When CAD-related reference values represent non-hyperemic indices (e.g., instantaneous wave-free ratio, resting whole-cycle ratio, diastolic non-hyperemic ratio, diastolic pressure ratio, resting Pd / Pa ratio), the output of the FFR drop regression also represents such non-hyperemic indices.

[0147] In an embodiment, during the training of a machine learning-based FFR pullback network (1706 in FIG. 17 and 1803 in FIG. 18), the predicted FFR drop can be monitored using the reference FFR drop along the axial trajectory of the coronary artery of interest. For this purpose, a new loss function inspired by the Earth Mover's Distance (EMD) and introducing a so-called EMD loss can be used. EMD is a method of measuring the overall similarity between distributions rather than locally comparing two probability distributions. Intuitively, EMD is the minimum amount of "work" required to transform one distribution into another. Therefore, "work" is defined as the amount of probability mass that needs to be moved multiplied by the distance it needs to be moved. Therefore, unlike point-by-point comparisons such as the mean absolute error (MAE), this loss function continuously increases along with the distance between the predicted FFR drop and the reference FFR drop. Thereby, if there is a misalignment between the predicted pullback curve and the reference pullback curve, the update of the gradient may be refined. In one dimension, the EMD between two probability distributions

Number

Number

Number

Number

Number

Number

Number

[0148] Equation 2 is used to calculate the loss between the predicted FFR drop and the reference FFR drop. Intuitively, this calculation corresponds to the cumulative difference between the FFR curves:

Number

Number

[0149] penalizes the proximal FFR drop difference more than the distal FFR drop difference. Instead, to treat the differences in FFR drops evenly regardless of location, we design a symmetric version of

Number

Number

Number

Number

Number

[0150] As an option, to be able to distinguish between focal FFR drop and chronic FFR drop, it is important that the predicted pullback curve drops with the same sharpness as the reference pullback curve. This is enforced by imposing a penalty on the difference between the histogram of the predicted FFR drop and the reference (so-called histogram loss is introduced). However, directly binning the output values makes differentiation impossible, so neural network training becomes impossible. To solve this problem, the Parzen-Rozenblatt window approach, which approximates the bins using a normal distribution with the mean located at the center of each bin, is used. The Parzen-Rozenblatt window method, as described in Non-Patent Document 32, does not require knowledge and assumptions about the underlying distribution and samples

Number

Number

Number

[0151] In an embodiment, a negative FFR drop may exist within a reference standard (1709). For example, such a negative FFR drop may result from an invasive pullback pressure measurement. As described in Non-Patent Document 33, such a negative FFR drop, which means an increase in FFR from proximal to distal, is mainly related to the hydrostatic effect. Optionally, in order to exclude the hydrostatic pressure, another loss function called Monotony loss can be included. By disabling the ReLU activation function as the last layer (see the description of step 1706 in FIG. 17), a negative FFR drop is allowed. These are suppressed by imposing a penalty on the sum of the negative FFR drops. Finally, at step 1707 in FIG. 17, an output (2006 in FIG. 20 and 2007 in FIG. 20) is provided. This output is data representing an FFR pullback along the axial trajectory of the relevant blood vessel.

[0152] FIG. 11 shows an example of a graphical user interface (display screen) that visually conveys the result of step 1507 to the user. 1102 in FIG. 11 shows a volume rendering image of CCTA data in which three major coronary arteries, the right coronary artery (RCA), the left anterior descending coronary artery (LAD), and the left circumflex coronary artery (LCX), are emphasized. The RCA is the coronary artery of interest, and this results in the multi-planar reconstruction (MPR) shown in part B of 1101 in FIG. 11. In this figure, the left side (1103) corresponds to the position of the opening (proximal) of the RCA, and the right side (1104) corresponds to the distal position of the RCA. Part C of 1101 shows a graph of the lumen area or lumen diameter along the RCA. The x-axis of this graph corresponds to the x-axis of the MPR view in part B of 1101 and the centerline of the RCA. Finally, part D of 1101 shows the FFR values along the centerline points of the RCA. The x-axis of this graph is the same as the x-axis of the MPR view and the diameter / area graph. Further, in 1102, the corresponding segments of the volume-rendered RCA are color-coded using the FFR values along the centerline of the volume-rendered RCA. It should be noted that the vertical markers in parts C and D of 1101 correspond to the minimum area / diameter (thick line marker) and the occlusion range (dashed line marker) obtained from general quantitative coronary analysis techniques.

[0153] As an option, in step 1710 of FIG. 17, different approaches are utilized to train a machine learning-based arterial characterization network (1704) and a machine learning-based FFR pullback network (1706). This “end-to-end training” is particularly valuable when the reference standard (1709) accommodates reference pullback values and single-point (distal) reference values (e.g., distal FFR and pullback FFR). First, both networks are trained simultaneously using the pullback reference values. Thereafter, the networks are trained individually (using the single-point reference values) or one of them is frozen. For example, the machine learning-based FFR pullback network is frozen and only the weights of the machine learning-based arterial characterization network are updated. In this example, only the loss of the machine learning-based FFR pullback network is used to calculate the loss. As a result, the machine learning-based arterial characterization network is optimized with respect to the final machine learning-based FFR pullback network. The advantage of this approach is that the machine learning-based arterial characterization network is optimized for the practical purpose of evaluating stenosis. This automatically results in focusing on the relevant segments (i.e., the segments where the lesion is present). Here, “focusing” specifically means that the contribution to the loss function, i.e., the contribution to weight updates, becomes larger for the lesion. This does not apply when monitoring the lumen area. In this case, segments with a large lumen area (segments less relevant to stenosis evaluation than segments with a small lumen area (i.e., lesions)) may result in the largest loss contribution.

[0154] Instead, the machine learning-based FFR pullback network is defined as a combination of a machine learning-based stenosis assessment network (206, FIG. 2) and a machine learning-based FFR pullback network (1706, FIG. 17). An example of such a combined network is illustrated in FIG. 21. Similar to the machine learning-based stenosis assessment network previously disclosed with reference to FIG. 2, the network of FIGS. 9, 10, 13, or 14 is composed of three stages. In the first stage (2101), the input as a result of the machine learning-based arterial characterization network (1704) and the coronary artery tree characteristics (1705) is sent to the encoder (2102) of the second stage, regardless of whether it is pre-encoded. The main difference is that the VAE encoding (from 1704) is used as an additional input (2104). In the third phase (2103), there are two output heads. These are the FFR drop regression head (2105) described with reference to FIG. 20 and the classification head (2106) described with reference to FIGS. 9, 10, 13, or 14. The outputs of the network illustrated in FIG. 21 are: 1) the predicted FFR pullback value (2106), 2) a single FFR value for each artery (2107), and 3) the prediction (2108) of the presence of a functionally significant stenosis by combining the output of the classification head (2109) and the output of the single FFR prediction (2107).

[0155] Furthermore, as described by Expansion 3 in the flowchart of FIG. 2, myocardial characteristics can be included in the flowchart of FIG. 17, and Expansion 3 the description of "including myocardial characteristics" also applies to the method described with reference to the flowchart of FIG. 17. FIG. 22 is an illustration for integrating myocardial characteristics into the machine learning-based FFR pullback network architecture of FIG. 21.

[0156] Experimental settings

[0157] The arterial characterization network (1704) was trained for 1200 epochs using MAE as the reconstruction loss, binary cross-entropy as the segmentation loss, and Kullback-Leibler information divergence for the regularization of the latent space (each weighted with a coefficient of 1). In particular, to refine the arterial representation, the reconstruction loss within the reference segmentation (lumen and plaque) is weighted with a coefficient of 5. To supervise the lumen area regression during training, MAE with a weight of 10 was used. The network was optimized using the ADAMW (Non-Patent Document 22) optimizer with a learning rate of 10 -5 and a batch size of 512. Once training was complete, we applied the network to each cross-section of the MPR and extracted the lumen area and VAE encoding along the centerline.

[0158] The FFR pullback network (1706) was trained for 150 epochs using the ADAMW optimizer with a cyclic learning rate with a linear schedule. The cyclic learning rate varied between 5e-4 and 1e-5 over a period of 40 epochs. Due to the limitation imposed by the different arterial lengths, the network would only be able to process one artery at a time. Thus, the loss was accumulated over 8 training iterations (equivalent to an effective batch size of 8) before backpropagation of the error. To monitor the FFR pullback, we pooled the output of the network to a step size of 2 mm using average pooling with a kernel size of 4 to offset the pullback reference fidelity and sensitivity to noise. To minimize misalignment, the invasive pullback measurement was manually registered in the input by shifting the start of the pullback signal so that the FFR drop optimally overlaps with the lumen stenosis. Since the pullback reference only covers a part of the artery, the pullback monitoring was only applied to that part by masking the distal overlap. The EMD loss and the histogram loss were weighted with coefficients of 0.1 and 5, respectively, for example. Optionally, the monotonicity loss was weighted with 20, for example. These coefficients were selected based on preliminary experiments to achieve a similar magnitude for the loss terms.

[0159] Here, referring to FIG. 23, another embodiment of the present application is disclosed. The steps shown here can of course be carried out in any logical order and can be partially omitted. The methods described in the flowcharts of FIGS. 2 and 17 first characterize the artery by lumen area and teacherless features, and then use these characteristics to predict the FFR pullback. This two-step approach allows for manual modification of the intermediate outputs, i.e., these characteristics. This not only helps to correct potential errors, but also allows for predicting the FFR pullback after successful percutaneous coronary intervention (PCI). PCI refers to a family of minimally invasive procedures used to open clogged coronary arteries (arteries that carry blood to the heart). By restoring blood flow, the treatment can refine the symptoms of blocked arteries such as chest pain and shortness of breath. PCI combines coronary angioplasty with stent placement (where a permanent wire mesh tube made of a drug-eluting or bare metal stent is inserted). Inflating the stent delivery balloon with a medium from an angioplasty catheter forces contact between the struts of the stent and the vessel wall (stent attachment) to widen the diameter of the blood vessel. This procedure visualizes the blood vessel by x-ray image using a coronary catheter after accessing the blood flow from the femoral artery or the radial artery. After that, an interventional cardiologist can perform coronary angioplasty using a balloon catheter. In this balloon catheter, the deflated balloon is advanced to the blocked artery and inflated to relieve the stenosis. Certain devices such as stents can be deployed to keep the blood vessel open. Various other procedures can also be carried out. FIG. 23 illustrates a flowchart for calculating the FFR pullback after successful PCI treatment using a CCTA image acquired before treatment. As described above, this workflow can also be used to incorporate manual adjustment of the vessel characteristics obtained from a machine learning-based artery characterization network (204 in FIG. 2 or 1704 in FIG. 17).

[0160] In step 2301 of FIG. 23, the method described in the flowchart of FIG. 2 or the method described in the flowchart of FIG. 17 is executed.

[0161] In step 2302 of FIG. 23, the derived arterial characteristics (204 in FIG. 2 or 1704 in FIG. 17) are adjusted. As described above, the arterial characteristics can be changed manually, semi-automatically or automatically. To simulate the success of PCI, the lumen area can be enlarged in the environment around the lesion to mimic the stent to be placed, and plaque components (calcified, non-calcified, mixed) can be removed from the treatment area. FIG. 24 is a diagram showing a method of changing arterial characteristics to simulate the success of PCI treatment. The cardiovascular MRI image is shown at 2401. Here, the virtual stent placement is illustrated at 2402 which also shows the lesion range. The cross-sectional image proximal to the lesion range is illustrated at 2403, and the cross-sectional image distal to the lesion range is illustrated at 2404. The cross-sectional slices will both represent the healthy cross-sectional area just outside the occlusion range. FIG. 24 shows two derived arterial characteristics (204 in FIG. 2 or 1704 in FIG. 17): the cross-sectional area (2405) along the axial trajectory of the cardiovascular vessel and the calcified plaque area (2406) along the axial trajectory of the cardiovascular vessel. By changing the cross-sectional area graph (2405) within the lesion range (2402) (as a result, a healthy cross-sectional area is obtained as shown at 2407), the success of PCI is simulated. All plaque components within the virtual stent placement shown at 2402 are removed because they will not obstruct blood flow after stent placement. For example, the calcified plaque area (2406) along the axial trajectory of the cardiovascular vessel, the portion within the virtual stent (2402), can be set to zero.

[0162] Next, a method is described for automatically defining a lesion segment and calculating a healthy reference area within the lesion segment. This will enable a successful simulation of the PCI procedure and the calculation of the FFR pullback after such a simulated PCI intervention. In FIG. 25, the cross-sectional area (2501) derived along the axial trajectory of the coronary artery, as a result of step 204 of FIG. 2 or step 1704 of FIG. 17, is visualized. The lesion range (2502), also referred to as the obstruction range or virtual stent range, is calculated, for example, from the position of a significant change in the curvature of the area graph (2501) covering the minimum area (2503), as described in non-patent document 34. Next, during fitting, as described in non-patent document 30, the area values within the lesion range (2402) are excluded and a straight line is fitted (2404) to pass through the area values along the axial trajectory of the coronary artery. The line (2504) represents the cross-sectional area along the axial trajectory of the coronary artery when the blood vessel is healthy and is also referred to as the reference area along the coronary artery. If there is a branch artery (branch) within the lesion range (defined in step 205 of FIG. 2 or step 1705 of FIG. 17), another method can also be used to simulate the step-down effect of the healthy coronary artery after the branch and calculate the reference area along the coronary artery. In FIG. 25, 2505 shows an example of a cross-sectional area graph (with a branch shown at 2506). The proximal reference area (2508) is calculated by extrapolating a line that fits from the proximal lesion range (2502) to the branch position (2506) (a line passing through the area values along the axial trajectory of the coronary artery, excluding all values from the proximal lesion range). Similarly, the distal reference area (2507) is calculated by extrapolating a line that fits from the distal lesion range (2402) to the branch position (2506) (a line passing through the area values along the axial trajectory of the coronary artery, excluding all values from the distal lesion range). Furthermore, Murray's law can also be incorporated into the calculation of the proximal reference area (2508) and the distal reference area (2507). Murray's law describes the relationship of the radii at the junctions of a network of tubular pipes carrying fluid. Radius

Number

Number

Number

Number

[0163] Next, referring to FIG. 26, another automated method is described for simulating virtual stent placement using the dense latent space of a variational autoencoder. In FIG. 26, a virtual stent placement is shown at 2602, and a cardiovascular MRI image is illustrated at 2601. A cross-sectional image proximal to the lesion extent is illustrated at 2603, and a cross-sectional image distal to the lesion extent is illustrated at 2604. The cross-sectional slices both represent the healthy cross-sectional area just outside the occlusion range. In the graph 2605 of FIG. 26, the cross-sectional area (2609) as a result of the arterial characterization network (1704 in FIG. 17) is illustrated. Further, an FFR pullback (2610) as a result of 1706 in FIG. 17 is shown.

[0164] As described above with reference to FIGS. 19a and 19b, a variational autoencoder can be trained to extract relevant information from a feature vector

Number

Number

Number

Number

Number

[0165] Finally, in step 2303 of FIG. 23, the FFR pullback after virtual stent placement is calculated. In step 2302, the vascular characteristics simulating the virtual stent placement have been calculated. The adjusted vascular characteristics are sent to the machine learning-based stenosis evaluation network of step 206 in FIG. 2, or the adjusted vascular characteristics are sent to the machine learning-based FFR pullback network of step 1706 in FIG. 17, and the FFR and / or FFR pullback simulating the virtual stent placement are calculated. An example of inputting the adjusted vascular characteristics (2302) into the machine learning-based FFR pullback network of step 1706 in FIG. 17 is illustrated in photograph 2606 of FIG. 26. The FFR pullback graph after virtual stent placement is shown at 2611. In this example, it can be seen that the FFR drop within the lesion segment (2602) has been eliminated as compared to the pre-treatment FFR pullback graph (2610).

[0166] Alternatively, without adjusting the vascular characteristics (2302), the virtual stent placement can be simulated and the MPR of the relevant blood vessel can be directly adjusted within the workflow of FIG. 2 or FIG. 17. First, as previously described with reference to FIG. 25, the extent of the lesion is determined from the lumen area graph. Next, the MPR slice covering the extent of the lesion within the MPR of the relevant blood vessel is determined. For example, referring to FIG. 26, photograph 2601 shows that a healthy MPR image is created along the axial trajectory of the relevant blood vessel by interpolating between the healthy cross-sectional MRI slices 2603 and 2604. Such interpolation can also be performed by a deep learning network trained to generate / interpolate new cross-sectional slices between a healthy proximal cross-sectional slice and a healthy distal cross-sectional slice from the MPR image. Finally, this new MPR of the relevant blood vessel is sent to the machine learning-based arterial characterization network (204 in FIG. 2 or 1704 in FIG. 17), and the remaining steps of FIG. 2 or FIG. 17 are executed.

[0167] Another embodiment of the present application is disclosed with reference to FIG. 28. The steps shown herein can clearly be executed in any logical order and can be partially omitted. FIG. 28 shows a flowchart of a machine learning-based method for automatically extracting a coronary artery tree and anatomically labeling the coronary artery tree.

[0168] FIG. 29 shows an overview of a machine learning-based method for automatically extracting a coronary artery tree in a CCTA image by automatically and repeatedly tracking the placed seed points. Thereafter, an ensemble of graph convolutional neutral networks (GCNs) is used to refine the extracted tree and label its segments. Referring to FIG. 29, after initialization (2901) by a seed (2905) and an inlet (2906), the coronary artery tree is extracted by 1) repeated tracking (2902), after which 2) an ensemble of GCNs is applied to refine the initially extracted tree (2903). Finally, another GCN is configured to label the anatomical segments (2904).

[0169] In step 2801 of FIG. 28, a CCTA image dataset is obtained. Such an image dataset represents a volumetric CCTA image dataset (e.g., a single contrast-enhanced CCTA dataset). This CCTA dataset can be obtained from a digital storage database such as a Picture Archiving and Communication System (PACS) or a VNA (Vendor Neutral Archive), a local digital storage database, a cloud database, or directly from a CT imaging modality. During CCTA imaging, a contrast agent is administered to the patient. Further, CCTA imaging can be triggered by an ECG.

[0170] The coronary artery tree is represented as an undirected tree graph. Each point on the centerline corresponds to a node of the graph, and the connections between the centerline points are represented by undirected edges. In step 2802 of FIG. 28, the processor initializes the tree graph. The tree graph is initialized by automatically placing seed points at the coronary artery (2905) and the left and right coronary artery inlets (2906) (2901). Thereafter, the tree graph is directly constructed during coronary artery extraction by simultaneously tracking the centerline (2907) of the coronary artery from the specified seed points. In this process, new points of the coronary artery are repeatedly added to the tree graph. In this way, tracking the section multiple times is avoided, and the redundancy of the calculation is reduced.

[0171] In the initialization, the positions of the seed points and the coronary artery inlets are predicted by two fully convolutional neural networks (seed CNN and inlet CNN). The architectures identical in both networks consist of seven 3D convolutional layers with a kernel width of 3. In layers 1 to 4, the number of channels is set to 32, and in layers 5 and 6 it is set to 64. In the last layer, a single output channel is generated. In order to expand the receptive field, dilation coefficients of 2 and 4 are used in layers 3 and 4 respectively, and the dilation is set to 1 in the remaining layers. The seed CNN and the inlet CNN are trained to predict, for each voxel, that the distance to the closest centerline or inlet of the coronary artery is a negative exponent. Thereby, a prediction map similar to a heat map indicating the positions of the coronary artery and the inlet is rendered. Thereafter, the seed points are identified as local maxima from the predicted heat map.

[0172] In step 2803 of FIG. 28, the processor performs tracking of the coronary artery tree. To add a new node to the tree, a CNN (tracking CNN) is used to predict the direction and step size from each endpoint of the graph. The tracking CNN receives a 3D image patch at the position of the seed point and predicts the direction and the radius of the coronary artery in the form of a binary output for discrete positions placed equidistantly on the unit sphere. To obtain a new point, the step proceeds in the predicted direction with a step size corresponding to the predicted blood vessel radius. To prevent prediction in the backward direction, the predicted direction classes close to the tracking direction are masked. The architecture of the tracking CNN is the same as that of the seed CNN and the inlet CNN (described in step 2802), except for the number of output nodes in the seventh layer. The tracking CNN uses 501 output channels (500 direction classes and one additional channel for radius regression). The output of the direction classes is then combined using a softmax activation layer.

[0173] Only endpoints, i.e., nodes with fewer than two edges, are tracked. Only those nodes among these with uncertainty in the direction prediction below a specific threshold are tracked. Nodes not connected to the inlet are less likely to be present in the coronary artery, so a different uncertainty threshold is applied to nodes connected to the inlet than to all other nodes. The uncertainty is given by the entropy of the direction output classes.

[0174] If all seed points are tracked simultaneously, multiple subgraphs are generated. Therefore, when these tracked subgraphs overlap, they are merged. An example of simultaneous tracking is provided in FIG. 30, which illustrates the tracking of two seed points, a blue node (3001) and a red node (3002). The two tracked subgraphs (blue and red nodes) overlap after three steps, so they are merged into one connected graph (green node, 3003) at step 4. It should be noted that this merge avoids duplicate tracking of proximal nodes on the branch (3004). Therefore, new nodes are created only if their location is not occupied by another node, i.e., if the node is not part of another subgraph. Node overlap is evaluated using the arterial radius of each existing node predicted by the tracker. FIGS. 31, 3101 show the construction of a tree graph including seed extraction and subsequent arterial tracking.

[0175] In step 2804 of FIG. 28, the coronary artery tree is refined. During the tree tracking described by step 2803, high sensitivity is archived to obtain a complete coronary artery tree (2902). However, this may come at the expense of a limited precision rate (i.e., false positive extraction representing veins or other tubular structures). Therefore, an ensemble of GCNs (2908) is used to remove false positives from the extracted tree. GCN is a generalization of CNN for any graph input. Therefore, using GCN enables direct learning of the combined representation of node features and connectivity from the graph. Therefore, the features that can be used by GCN later need to be defined.

[0176] To enable refinement of the initially extracted tree, arterial segments are created by grouping adjacent centerline points of the tree graph. These segments are characterized using a set of features that describe the location, direction, shape, and appearance of the segment in the image.

[0177] The position of the segment is described, for example, as in Non-Patent Document 35, by the orthogonal coordinates of the center line points of the segment at the start, end, and quartiles of the segment length, with reference to the center of the left ventricular myocardium. Two features for describing the direction of the segment are extracted. The first feature corresponds to the orthogonal coordinates of the normalized direction vector between the end points of the segment. The second feature is constituted by the orthogonal coordinates of the normalized direction vector between the first two center line points of the segment. To describe the shape of the blood vessel, the average and standard deviation of the blood vessel radius of the center line points within the segment, corresponding to the size of the blood vessel, are used.

[0178] The appearance of the segment is characterized by the texture derived using the outputs of the tracking CNN (described by 2907 and step 2803) and the seed CNN (described by 2905 and step 2802). This characterization is described by the average and standard deviation of the entropy values of the center line points of the segment. The entropy value corresponds to the uncertainty of the tracker at that location. Intuitively, this indicates the degree to which the segment resembles a blood vessel. Similarly, the average and standard deviation of the output values of the seed CNN are calculated and extracted from the output map at the center line point positions within the segment.

[0179] To distinguish actual coronary artery segments (positive class) from structures such as other blood vessels (negative class), binary classification is performed using a GCN. The GCN enables learning of the combined representation of node features and connections by directly using the tree graph initially extracted as the input for classification. To enhance the robustness against potentially missing segments, an ensemble of GCNs is used and applied to graphs of multiple different resolutions.

[0180] In a graph attention network (GAT), the weights for aggregating features from adjacent nodes are learned end-to-end using an attention subnetwork. The aggregation function of the GAT is

Number

Number

Number

Number

Number

Number

Number

[0181] To utilize information regarding the geometric structure of the coronary artery tree, a sufficiently large receptive field is required. However, as described in Non-Patent Document 36, the receptive field of GCN is usually limited. Therefore, the effective receptive field is increased by using a coarse graph with low resolution as the input. To create the coarse graph, the adjacent nodes of the initially extracted tree (fine graph) are grouped into segments.

[0182] When nodes are grouped, the receptive field expands, but it is also possible that true and false nodes are grouped into the same segment (e.g., arteries leak into veins, resulting in label ambiguity) (see FIGS. 32 and 3201). Therefore, multiple coarse graphs (3202) are created by grouping the centerline nodes into segments in two different ways. For the first coarse graph, we apply the conventional definition taught by Non-Patent Document 37, where segments are separated by branches. To enable the subdivision of these segments, additional graphs are created by dividing the segments of the above-mentioned undivided graph into small segments with low resolution. To enable information flow between all segments adjacent to the branches, the edges of the graph are established by combining all possible pairs of the relevant segments. Furthermore, the information flow from each node to itself becomes effective by inserting an edge from each node to itself.

[0183] To combine the advantages of reduced label ambiguity (graph of small segments) and large receptive fields (graph of large segments), a multi-resolution graph ensemble strategy is adopted. For this purpose, the predictions of all graphs with different resolutions are back-projected onto the fine-grained graph. The ensemble was performed by taking the average of the output probabilities of all GCNs on the fine-grained graph.

[0184] The GCNs within the ensemble are applied to graphs with different resolutions, but the network architecture schematically shown in FIG. 33 is the same for all GCNs. It consists of three GAT layers with dense connections from the input features to all successive GAT layers. In each GAT layer, four heads with eight encodings per head are used in combination with residual connections and dropout. Additionally, the input features are concatenated to the input of each successive GAT layer. Finally, binary output probabilities corresponding to the actual segments of structures such as the coronary artery tree and other blood vessels are predicted using a linear layer followed by a softmax activation function. FIG. 3102 in FIG. 31 shows the result of the tree refinement step.

[0185] Finally, in step 2805 of FIG. 28, labeling of the anatomical coronary arteries is performed. The anatomical labels are assigned to segments of the extracted coronary artery tree (2904). Similar to tree extraction, an ensemble of GCNs (2909) trained on graphs of different resolutions is employed for anatomical labeling. Labeling of segments is posed as a multi-class classification task. By using GCNs, this model can learn information about individual segments and the relationships between these segments. Furthermore, the multi-resolution ensemble approach reduces problems associated with label ambiguity due to missing branches. The receptive field guaranteed by this method is large enough to model the relationships between segments. Graphs of different resolutions are obtained by grouping centerline points into segments in the same way as described in step 2804. The network architecture for anatomical labeling is the same as that for tree refinement (step 2804), except that in anatomical labeling, a total of 10 output classes are used, one for each coronary artery segment label (from the ground truth 2806). Furthermore, the same input features are employed. FIG. 34 illustrates the results of step 2805. This figure shows the extracted coronary artery tree with each coronary artery segment anatomically labeled.

[0186] The ground truth (2806) is a database that accommodates data of multiple patients. Each set within the database contains, for each patient, a) a contrast CT image dataset (2801 represents the reference image set during the training phase) and the corresponding coronary artery centerline tree in which, for example, according to the model introduced by Non-Patent Document 38, the lumen radius and anatomical labels are assigned to each centerline point within the tree.

[0187] Experimental Setup

[0188] All GCNs were trained using the Adam optimizer over 500 epochs using the reference standard (2806). Unless otherwise specified, the learning rate is set to 0.001 for the first 300 epochs and 0.0001 for the remaining 200 epochs. A probability of 0.2 is used for dropout.

[0189] Initialization of the tree and tree tracking (2802, 2803): All three CNNs (2905, 2906, 2907) are trained using the reference standard (2806) described in Non-Patent Document 39. For the initialization of tracking, 400 seed points are used. We selected such a large number of seed points to ensure high sensitivity. Furthermore, we generated the seed points so that they do not get too close to each other by reducing redundancy and setting the minimum distance between the seed points to 3 mm. In the case of arterial tracking, the input to the tracking CNN was a 19x19x19 image patch, and the number of output directions was 500. We tracked the nodes connected to the inlet when the entropy of the output direction (i.e., uncertainty) was less than 0.9 and the nodes disconnected from the inlet when the entropy was less than 0.7. To avoid backward tracking, we masked the directions less than 60 degrees from the previously tracked direction. The tracking is terminated when there are no remaining nodes to track or when the maximum number of 40 steps determined in the preliminary experiment is reached (this prevents the extraction of extensive false detections). In the proposed graph tracking, the number of tracked nodes is reduced from an average of 7,667 to 4,476 (42%) compared to the case of redundantly tracking the proximal section. After the tracking is completed, we deleted all single nodes (since they presented seed points that were not tracked). If the extracted seed points become the starting points of tree construction, leaks due to inaccurate seed points and inaccurate prediction of the centerline direction may cause false positives (typically, artery-like structures such as veins). Photographs 3501 and 3502 in FIG. 35 illustrate that the seed CNN outputs high values for the coronary artery and low values for the coronary vein. Photograph 3501 shows an axial slice of the CCTA image dataset in which the coronary artery (3503) and vein (3504) are shown side by side. Photograph 3502 shows the output map of the seed CNN indicating high probability values for the coronary artery and low probability values for the coronary vein.Since the tracker is trained on relatively small image patches around the centerline voxel, it tracks the coronary artery (3505), but also tracks other vessel-like structures such as the coronary vein (3506). Photograph 3505 shows the output of the tracking CNN for a patch around the coronary artery seed point (the position in the CCTA is marked by circle 3503). The probability output is visualized as an activation on the unit sphere (the size of the activation corresponds to the magnitude of the probability). Photograph 3506 shows the output of the tracking CNN for a patch around the coronary vein seed point (the position in the CCTA is marked by the dashed circle 3504). Thus, after tree tracking, we refined the tree (2804) by removing false positives.

[0190] Refinement of the tree (2804) : To learn from large receptive fields while reducing label ambiguity (Figure 32), we used a graph ensemble strategy based on coarse graphs of different resolutions. Specifically, in addition to the unsegmented graph, we created graphs with a predefined centerline point (e.g., 10) for each segment.

[0191] Anatomical labeling (2805) : To learn the robustness of anatomical labeling against potential errors from tree extraction, we used the automatically extracted tree (the result of step 2804) as the input to the GCN for training instead of the reference tree from the ground truth (2806). The automatically extracted tree was represented in the same coarse graph as in the case of tree extraction for training the ensemble for anatomical labeling. Multiple graphs are preferably created for each segment using predefined centerline points such as, for example, 5, 10, 20, 30. Thus, to set the reference for training the GCN, we projected the anatomical labels from the reference tree onto the automatically extracted coarse graph. During the training of anatomical labeling, nodes that exist in the automatically extracted tree but not in the reference tree (false positive nodes) were not processed by the error backpropagation method.

[0192] In the present disclosure, primarily, the organ of interest is described as the myocardium, and the blood vessels as the coronary arteries. Those skilled in the art will understand that this teaching can be similarly extended to other organs. For example, the organ of interest can be the kidney perfused by the renal artery, or the brain (a part of it) perfused by the intracranial artery. Further, the present disclosure refers to (some form of) CCTA datasets. Those skilled in the art will understand that this teaching can be similarly extended to other imaging diagnostic methods (such as rotational angiography, MRI, SPECT, PET, ultrasound, x-ray, etc.).

[0193] Embodiments of the present disclosure can be used in a stand-alone system or, for example, can also be directly incorporated into a computed tomography (CT) system. FIG. 36 illustrates an example of a high-level block diagram of a computed tomography (CT) system. This block diagram includes an example showing how an embodiment is integrated into such a system.

[0194] A part of the system (defined by various functional blocks) may be implemented by dedicated hardware, analog and / or digital circuits, and / or one or more processors that execute program instructions stored in memory.

[0195] The most common computed tomography method is x-ray CT, but there are also many other types of CT, such as dual-energy, spectral multi-energy, photon-counting CT. Also, this can be combined with positron emission tomography (PET) and single-photon emission computed tomography (SPECT), or previous forms of CT.

[0196] The CT system of FIG. 36 describes an x-ray CT system. In an x-ray CT system, the x-ray system moves around the patient within the gantry and obtains images. By using digital processing, a three-dimensional image is constructed from a series of two-dimensional angiography images taken around a single axis of rotation.

[0197] In a general x-ray CT system 120, an operator places a patient 1200 on a patient table 1201 and uses an operation console 1202 to provide scan inputs. The operation console 1202 typically consists of a computer, a keyboard / foot pedal / touch screen, and one or more monitors.

[0198] The operation control computer 1203 not only uses the inputs from the operation console to command the gantry 1204 to rotate, but also sends commands to execute scans to the patient table 1201 and the x-ray system 1205.

[0199] The operation control computer 1203 uses the scan protocol selected on the operation console 1202 to send a series of commands to the gantry 1204, the patient table 1201, and the x-ray system 1205. After that, the gantry 1204 reaches a certain rotation speed and maintains that speed throughout the scan. The patient table 1201 reaches the intended starting position and maintains a constant speed throughout the scan process.

[0200] The x-ray system 1205 includes an x-ray tube 1206 with a high-voltage generator 1207 that generates an x-ray beam 1208.

[0201] The high-voltage generator 1207 controls and powers the x-ray tube 1206. The high-voltage generator 1207 applies a high voltage across the vacuum gap between the cathode and the rotating anode of the x-ray tube 1206.

[0202] The voltage applied to the x-ray tube 1206 causes electron movement from the cathode to the anode of the x-ray tube 1206, resulting in an x-ray photon generation effect, also known as bremsstrahlung. The generated photons form an x-ray beam 1208 directed towards the image detector 1209.

[0203] The x-ray beam 1208 consists of photons with an energy spectrum in the range up to a maximum value determined by the voltage, current, etc. supplied to the x-ray tube 1206.

[0204] The x-ray beam 1208 then passes through the patient 1200 lying on the moving table 1201. The x-ray photons of the x-ray beam 1208 penetrate the patient's tissue to varying degrees. The various structures of the patient 1200 absorb the radiation at various rates and modulate the intensity of the beam.

[0205] The modulated x-ray beam 1208` emerging from the patient 1200 is detected by the image detector 1209 located on the opposite side of the x-ray tube.

[0206] This image detector 1209 can be either an indirect detection system or a direct detection system.

[0207] In the case of an indirect detection system, the image detector 1209 comprises a vacuum tube (x-ray image intensifier) that converts the x-ray output beam 1208` into an amplified visible light image.

[0208] This amplified visible light image is transmitted to a visible light image receptor such as a digital video camera for image display and recording. Thereby, a digital image signal is generated.

[0209] In the case of a direct detection system, the image detector 1209 comprises a flat panel detector. The flat panel detector directly converts the x-ray exit beam 1208` into a digital image signal.

[0210] The digital image signal generated from the image detector 1209 is passed to the image generator 1210 for processing. Usually, the image generation system houses a high-speed computer and digital signal processing chips. The acquired data is pre-processed and enhanced before being transmitted to the display device 1202 for the operator to view and to the data storage device 1211 for archiving.

[0211] The gantry has the x-ray system arranged such that the patient 1200 and the moving table 1201 are positioned between the x-ray tube 1206 and the image detector 1209.

[0212] In contrast-enhanced CT scanning, the injection of the contrast agent needs to be synchronized with the scan. The contrast agent injector 1212 is controlled by the operation control computer 1203.

[0213] For FFR measurement, there is an FFR guide wire 1213, and in order to induce the maximum hyperemic state, adenosine is injected into the patient by an injector 1214.

[0214] Embodiments of the present application are implemented as follows by the x-ray CT system 120 of FIG. 18. A clinician or other user selects a scan protocol using the operation console 1202 to obtain a CT scan of the patient 1200. The patient 1200 lies on an adjustable table 1201 that moves at a continuous speed throughout the scan and is controlled by the operation control computer 1203. The gantry 1204 maintains a constant rotational speed throughout the scan.

[0215] Next, a plurality of two-dimensional x-ray images are generated using the high-voltage generator 1207, the x-ray tube 1206, the image detector 1209, and the digital image generator 1210 described above. This image is stored in the hard drive 1211. Using these x-ray images, a three-dimensional image is constructed by the image generator 1210.

[0216] The general-purpose processing unit 1215 uses the three-dimensional image to perform classification as described above.

[0217] Some embodiments of methods and apparatuses for automatically identifying patients having functionally significant stenosis based on information extracted from only a single CCTA image have been described and illustrated herein.

[0218] Particular embodiments of the present application have been described, but the present application is not intended to be limited thereto. The present application is directed to as broad a scope as the technology permits, and this specification is intended to be interpreted accordingly.

[0219] For example, a multi-phase CCTA dataset can be used, and the functional evaluation of the renal artery for a perfused kidney can be evaluated based on the disclosed methodology. The data processing operations may be performed offline on images stored in digital storage, such as a PACS or VNA in DICOM (Digital Imaging and Communications in Medicine) format, which is commonly used in medical imaging technology. Therefore, those skilled in the art will understand that other changes can be made to the provided application without departing from the claimed spirit and scope.

[0220] As described above, the embodiments described herein may include various data stores and other memories and storage media. These can exist in various locations, such as on the local (and / or resident) storage media of one or more computers, or at a remote location from any or all of the computers across the network. In certain embodiments, the information may exist in a storage area network ("SAN: Storage-Area Network") well-known to those skilled in the art.

[0221] Similarly, the files necessary to perform the functions attributed to a computer, server, or other network device may be stored locally and / or remotely as needed.

[0222] If the system includes computerized devices, each such device can include hardware elements that may be electrically coupled via a bus. These elements can include, for example, at least one central processing unit (“CPU” or “processor”), at least one input device (e.g., mouse, keyboard, controller, touch screen, keypad), and at least one output device (e.g., display device, printer, speaker). Such a system may include one or more storage devices such as a disk drive, an optical storage device, solid state storage devices such as random access memory (“RAM”) and read only memory (“ROM”), removable media devices, memory cards, flash cards.

[0223] As described above, such devices may also include a computer-readable storage media reader, a communication device (modem, network card (wireless or wired), infrared communication device, etc.), and a working memory.

[0224] The computer-readable storage media reader can be configured to connect to, or receive, a computer-readable storage media representing a remote, local, fixed, and / or removable storage device, and a storage media for temporarily and / or more permanently accommodating, storing, transmitting, and acquiring computer-readable information. The system and various devices will typically include a number of software applications, modules, services, or other elements disposed within at least one working memory device, including an operating system and application programs such as client applications or a web browser.

[0225] It should be understood that alternative embodiments can have various variations from the above description. For example, customized hardware may also be used, and / or certain elements may be implemented in hardware, software (including portable software such as applets), or both.

[0226] Furthermore, connections to other computing devices such as network input / output devices may be employed.

[0227] Various embodiments may further include receiving, transmitting, or storing instructions and / or data implemented according to the foregoing description on a computer-readable medium. Storage media and computer-readable media for containing code or a portion of code may be any suitable media known or used in the art, including (but not limited to) volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information such as computer-readable instructions, data structures, program modules, or other data. This media can include RAM, ROM, electrically erasable programmable read-only memory ("EEPROM"), flash memory or other memory technologies, compact disc read-only memory ("CD-ROM"), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other media that can be used to store the required information and can be accessed by a system device. Based on the disclosure and teachings provided herein, those skilled in the art will understand other means and / or methods for implementing various embodiments.

[0228] FIG. 37 shows a diagrammatic representation of a machine in the form of an example of a computer system 18000 capable of executing a set of instructions for causing the machine to execute any one or more of the methods, processes, operations, or methodologies described herein. In alternative embodiments, the machine may operate as a stand-alone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate as a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a cellular phone, a smart phone, a web appliance, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute a set of instructions (or multiple sets) to perform one or more of the methodologies described herein. Also, exemplary embodiments may be implemented in a distributed system environment having local and remote computer systems that are linked (e.g., by any one of a wired, wireless, or a combination of wired and wireless connections) through a network. In a distributed system environment, program modules may be located in both local and remote memory storage devices (see below).

[0229] The exemplary computer system 18000 includes a processor 18002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 18001, and a static memory 18006, which communicate with each other via a bus 18008. The computer system 18000 may further include a video display unit 18010 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system 18000 also includes an alphanumeric input device 18012 (e.g., a keyboard), a user interface (UI) cursor controller 18014 (e.g., a mouse), a disk drive unit 18016, a signal generation device 18018 (e.g., a speaker), and a network interface device 18020 (e.g., a transmitter).

[0230] The disk drive unit 18016 includes a machine-readable medium 18022 storing one or more instruction sets 18024 and data structures (software) that embody or use one or more of the methodologies or functions described herein. The software may reside, completely or at least partially, within the main memory 18001 and / or within the processor 18002 while being executed by the computer system 18000 (the main memory 18001 and the processor 18002 also constitute machine-readable media).

[0231] Action 18024 may further cause transmission or reception via the network 18026 using any of a number of well-known transfer protocols (e.g., HTTP, Session Initiation Protocol (SIP)) via the network interface device 18020.

[0232] The term "machine-readable medium" should be interpreted to include a single medium or a plurality of media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more instruction sets. Also, the term "machine-readable medium" should be interpreted to include any medium that can store, encode, or carry an instruction set for execution by a machine and that can cause the machine to perform any of the one or more methods described herein. Thus, the term "machine-readable medium" is interpreted to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0233] The method embodiments described herein may be implemented on a computer. Some embodiments may include a computer-readable medium encoded with a computer program (e.g., software) that includes executable instructions for causing an electronic device to perform the methods of the various embodiments. The software implementation (or computer-implemented method) may include microcode, assembly language code, or high-level language code (which may further include computer-readable instructions for performing the various methods). The code may form part of a computer program product. Further, the code may be tangibly stored on one or more volatile or non-volatile computer-readable media during execution or at other times. These computer-readable media may include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memory (RAM), read-only memory (ROM), etc.

[0234] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. However, it will be apparent that various modifications and changes may be made to these without departing from the broader spirit and scope of the application as claimed.

[0235] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative configurations, its particular embodiment shown in the drawings has been shown and described in detail above. However, it is to be understood that the intention is not to limit the present application to the particular forms disclosed, but on the contrary, to cover all modifications, alternative configurations, and equivalents included within the spirit and scope of the present application as defined by the appended claims.

[0236] The use of the terms "a" and "an" and "the" and similar references in the context of describing the disclosed embodiments (particularly in the context of the following claims) should be construed to cover both the singular and the plural forms unless otherwise indicated herein or clearly contradicted by the context. The terms "comprising", "having", "including", and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to") unless specifically stated otherwise. The term "connected", when not modified and referring to a physical connection, is to be construed as including being incorporated, attached, or joined, in whole or in part, with intervening elements.

[0237] The recitation of a range of values herein is intended solely as a shorthand method of referring individually to each value within the range, unless otherwise indicated herein and unless each individual value is incorporated into the specification as if it were individually recited herein. The use of the term "set" (e.g., "a set of items") or "subset", unless otherwise specified or inconsistent with the context, is to be construed as a non-empty collection containing one or more members.

[0238] Furthermore, unless otherwise specified or inconsistent with the context, the term "subset" of a corresponding set does not necessarily indicate a proper subset of the corresponding set, and the subset and the corresponding set may be equal.

[0239] The operations of the processes described herein can be performed in any suitable order, unless otherwise specified herein or clearly inconsistent with the context. The processes (or variations and / or combinations thereof) described herein may be performed under the control of one or more computer systems configured by executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed collectively by one or more processors, hardware, or combinations thereof. The code may be stored, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors, on a computer-readable storage medium. The computer-readable storage medium may be non-transitory.

[0240] Preferred embodiments of the present disclosure are described herein, including the best mode known to the inventors for carrying out the present application. Variations of these preferred embodiments will be apparent to those skilled in the art upon reading the foregoing description. The inventors expect those skilled in the art to appropriately employ such variations, and the inventors intend for the embodiments of the present disclosure to be practiced in ways other than those specifically described herein.

[0241] Accordingly, the scope of the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto, to the extent permitted by applicable law. Further, any combination of any possible variations of the above elements is included in the scope of the present disclosure, unless otherwise indicated herein or clearly inconsistent with the context.

[0242] Additional references are provided below: · Wong DTL, Ko BS, Cameron JD, Nerlekar N, Leung MCH, Malaiapan Y, et al., "Translumenal attenuation gradient in coronary CT angiography: a new non-invasive approach to identify functionally significant coronary stenosis: comparison with fractional flow reserve", J Am Coll Cardiol. (2013)61:1271-9. doi: 10.1016 / j.jacc.2012.12.029 · Ko BS, Wong DTL, Norgaard BL, Leong DP, Cameron JD, Gaur S, et al., "Diagnostic performance of trans-lumenal attenuation gradient and non-invasive fractional flow reserve derived from 320-detector row CT angiography in the diagnosis of hemodynamically significant coronary stenosis: the NXT substudy", Radiology. (2016)279:75-83. doi: 10.1148 / radiol.2015150383 · Diaz-Zamudio M, Dey D, Schuhbaeck A, Nakazato R, Gransar H, Slomka PJ, et al., "Automated quantification of plaque burden from coronary CT angiography predicts hemodynamic significance non-invasively using fractional flow reserve in intermediate coronary lesions", Radiology. (2015)276:408-15. doi: 10.1148 / radiol.2015141648 · Otaki Y, Han D, Klein E, Gransar H, Park RH, Tamarappoo B, et al., "Value of semi-quantitative assessment of high-risk plaque features on coronary CT angiography for stenosis in the selection of FFRct testing", J Cardiovasc Comput Tomogr. (2021)16:27-33. doi: 10.1016 / j.jcct.2021.06.004 · Gould KL, Lipscomb K, Calvert C., "Compensatory changes in the distal coronary vascular bed during progressive coronary stenosis", Circulation. (1975)51:1085-94. doi: 10.1161 / 01.CIR.51.6.1085 · Dey D, Achenbach S, Schuhbaeck A, Pflederer T, Nakazato R, Slomka PJ, et al., "Comparison of Quantitative Atherosclerotic Plaque Burden by Coronary CT Angiography in Patients with Acute Coronary Syndrome and Stable Coronary Artery Disease at First Presentation", J Cardiovasc Comput Tomogr. (2014)8:368 - 74. doi: 10.1016 / j.jcct.2014.07.007 · Hell MM, Dey D, Marwan M, Achenbach S, Schmid J, Schuhbaeck A., "Non - invasive Prediction of Hemodynamically Significant Coronary Stenosis by Contrast Concentration Difference in Coronary CT Angiography", Eur J Radiol. (2015)84:1502 - 8. doi: 10.1016 / j.ejrad.2015.04.024 · Ko BS, Wong DTL, Cameron JD, Leong DP, Soh S, Nerlekar N, et al., "ASLA Score: A CT Angiography Index for Predicting Functionally Significant Coronary Stenosis in Lesions of Moderate Severity - Diagnostic Accuracy", Radiology. (2015)276:91 - 101. doi: 10.1148 / radiol.15141231 · Dey D, Gaur S, Ovrehus KA, Slomka PJ, Betancur J, Goeller M, et al., "Integrated Prediction of Lesion - specific Ischemia from Quantitative Coronary CT Angiography Using Machine Learning: A Multicenter Study", Eur Radiol. (2018)28:2655 - 64. doi: 10.1007 / s00330 - 017 - 5223 - z · Yang S, Koo BK, Hoshino M, Lee JM, Murai T, Park J, et al., "CT Angiography and Plaque Predictors of Functionally Significant Coronary Disease and Outcomes Using Machine Learning", JACC Cardiovascular imaging. (2021)14:629 - 41. doi: 10.1016 / j.jcmg.2020.08.025 · Ghanem AM, Hamimi AH, Matta JR, Carass A, Elgarf RM, Gharib AM, "Automatic segmentation of coronary artery wall and atherosclerotic plaque from 3D coronary CT angiography", Sci Rep. (2019)9:47. doi: 10.1038 / s41598-018-37168-4 · Loshchilov, I., Hutter, F., 2019., "Regularization of decoupled weight decay", International Conference on Learning Representations - ICLR 2019

[0243] All references (including publications, patent applications, and patents) cited herein are hereby incorporated by reference in their entirety and to the same extent as if each reference had been individually and specifically indicated to be incorporated by reference and were set forth in full herein.

[0244] Enumerated clauses:

[0245] The enumerated clauses are provided for the purpose of exemplifying some of the embodiments that may be provided in accordance with the disclosure. The set of clauses shown below is for illustrative purposes and should not be construed as limiting, exclusive, or exhaustive. Features recited in one set of clauses may be utilized and incorporated in one or more other sets of clauses. In any one or more of the following sets of clauses, an embodiment may provide a computer-implemented method.

[0246] Clause set A1:

[0247] The embodiments disclosed herein can provide a method and system for evaluating a patient's cardiovascular occlusion, which may include one, some, or all of the following operations: Obtaining the volume image dataset of the cardiovascular system; Steps of analyzing the volume image data set to extract data representing the axial trajectory of the coronary blood vessel; Steps of generating a multi-planar reformation (MPR) image based on the volume image data set and the data representing the axial trajectory of the coronary blood vessel; Steps of generating first feature data characterizing the presence of zero or more branches or side branches along the axial trajectory of the coronary blood vessel; Steps of supplying the MPR image and the first feature data to a first machine learning network that outputs: i) a plurality of latent space encodings characterizing the features of the coronary blood vessel along the axial trajectory of the coronary blood vessel when the MPR image and the first feature data are provided; and ii) additional data characterizing additional features of the coronary blood vessel along the axial trajectory of the coronary blood vessel; and Steps of supplying the plurality of latent space encodings and the additional features output by the first machine learning network to a second machine learning network that outputs data characterizing the pullback of the fractional flow reserve (FFR) of the coronary blood vessel when the input data is provided.

[0248] A2. The method according to clause A1, further comprising displaying or outputting data characterizing the FFR pullback of the coronary blood vessel.

[0249] A3. The first feature data is generated from the analysis of the MPR image; and / or The first feature data is generated from the analysis of the volume image data set; and / or The first feature data is generated from a coronary centerline tree obtained from the volume image data set, The method according to clause A1 or A2.

[0250] A4. The method according to clause A1, A2 or A3, wherein the additional features characterized by the additional feature data output by the first machine learning network include at least one feature related to the lumen characteristics of the cardiovascular vessel (such as lumen area and / or lumen attenuation) along the axial trajectory of the cardiovascular vessel.

[0251] A5. The method according to clause A1, A2 or A3, wherein the additional features characterized by the additional feature data output by the first machine learning network include at least one feature related to the plaque characteristics of the cardiovascular vessel (such as calcium plaque area, soft plaque area, mixed plaque area) along the axial trajectory of the cardiovascular vessel.

[0252] A6. Furthermore, the method according to any one of clauses A1 - A5, including generating myocardial feature data characterizing a local portion of the myocardium related to the cardiovascular vessel, and supplying the myocardial feature data as an input to a second machine learning network for use in generating the data characterizing the FFR pullback of the cardiovascular vessel.

[0253] A7. The method according to any one of clauses A1 - A6, wherein the first machine learning network comprises a variational auto - encoder having an encoder portion that generates the plurality of latent space encodings, and the encoder portion is trained using unsupervised learning.

[0254] A8. The method according to clause A7, wherein the first feature data is input into the convolutional block of the encoder portion.

[0255] A9. The method according to clause A7, comprising at least one auxiliary decoder unit, wherein the first machine learning network is configured to generate the additional feature data when a subset of the latent space encoding generated by the encoder unit is supplied and the subset of the latent space encoding is given as an input.

[0256] A10. The method according to clause A7, wherein the at least one auxiliary decoder unit is trained by supervised learning using training data including reference annotations based on measurements or extractions of corresponding features of a plurality of patients.

[0257] A11. The method according to any one of clauses A1 to A10, wherein the second machine learning network is trained by supervised learning using training data including reference annotations based on measurements of FFR pullback values or FFR drop values associated with vascular centerline points of a plurality of patients.

[0258] A12. The method according to any one of clauses A1 to A11, wherein the second machine learning network is further configured to output an FFR value of a blood vessel, and the second machine is trained by supervised learning using training data including reference annotations based on measured values of FFR values associated with blood vessels of a plurality of patients.

[0259] A13. The second machine learning network is further configured to output data representing a prediction of the presence of a functionally significant stenosis, and the second machine learning network is trained by supervised learning using training data including reference annotations representing the presence of a functionally significant stenosis for a plurality of patients, The method according to any one of clauses A1 to A12.

[0260] A14. The method according to any one of clauses A1 to A13, wherein the second machine learning network includes a convolutional neural network, and the convolutional neural network is trained by supervised learning using training data including reference annotations for the output data of the second machine learning network.

[0261] A15. The method according to clause A14, wherein the reference annotation is derived by manual segmentation of the corresponding volume image data and / or automatic segmentation of the corresponding volume image data.

[0262] A16. The method according to clause A14, wherein the convolutional neural network of the second machine learning system includes a regression head that generates an FFR drop along the axial trajectory of the coronary vessel, and an output stage that generates the FFR pullback output by the second machine learning system.

[0263] A17. The method according to clause A14, wherein the convolutional neural network of the second machine learning system further includes a first classification head that outputs data representing the FFR value of a blood vessel.

[0264] A18. The method according to clause A14, wherein the convolutional neural network of the second machine learning system further includes a second classification head that outputs data representing the prediction of the presence of a functionally significant stenosis.

[0265] A19. The method according to any one of clauses A1 to A18, wherein the coronary vessel includes a coronary artery or a coronary artery tree.

[0266] A20. The method according to any one of clauses A1 to A19, wherein the volume image data set includes CCTA image data.

[0267] A21. A system for evaluating a patient's coronary vascular occlusion, comprising at least one processor configured to perform some or all of the operations of clauses A1 - A20 when executing program instructions stored in a memory.

[0268] A22. The system according to clause A21, further comprising an image acquisition subsystem configured to acquire the volume image data set.

[0269] A23. The system according to clause A22, further comprising a display subsystem configured to display the data characterizing the severity of the anatomical lesion of the coronary blood vessels.

[0270] A24. A non - transitory program storage device tangibly embodying a program of instructions executable on a machine for performing any or some of the operations of clauses A1 - A20 to evaluate a patient's coronary vascular occlusion.

[0271] Clause set B1:

[0272] The embodiments disclosed herein can provide methods and systems for simulating or planning interventional treatment of a patient's coronary vascular occlusion, which include one, some, or all of the following operations: The operation of obtaining the volume image data set of the coronary blood vessels; The operation of analyzing the volume image data set to extract data representing the axial trajectory of the coronary blood vessels; The operation of generating a multi - planar re - construction (MPR) image based on the volume image data set and the data representing the axial trajectory of the coronary blood vessels; Supplying the MPR image to a first machine learning network to output: i) a plurality of latent space encodings that characterize the features of the coronary vasculature along the axial trajectory of the coronary vasculature when the MPR image is provided; and ii) additional feature data that characterizes additional features of the coronary vasculature along the axial trajectory of the coronary vasculature; Adjusting some or all of the additional feature data based on a simulated or planned treatment of the coronary vasculature; Supplying the plurality of latent space encodings output by the first machine learning network and the adjusted additional feature data to a second machine learning network to output data that characterizes the FFR pullback of the coronary vasculature considering the simulated or planned treatment of the coronary vasculature when the input data is provided.

[0273] The operation of Clause B1 can follow any one or part of the operations of Clauses A1 - A20 above.

[0274] Clause Set B2:

[0275] The embodiments disclosed herein can provide a method and system for simulating or planning an intervention treatment for an occlusion of a patient's coronary vasculature, which includes one, some, or all of the following operations: Obtaining a volume image dataset of the coronary vasculature; Analyzing the volume image dataset to extract data representing the axial trajectory of the coronary vasculature; Generating a multi - planar reformation (MPR) image based on the volume image dataset and the data representing the axial trajectory of the coronary vasculature; Adjusting the MRP image based on a simulated or planned treatment of the coronary vasculature; Supplying the adjusted MPRG image to a first machine learning network to output: i) a plurality of latent space encodings characterizing the features of the coronary blood vessel along the axial trajectory of the coronary blood vessel when an MPR image is provided; and ii) additional feature data characterizing additional features of the coronary blood vessel along the axial trajectory of the coronary blood vessel; and Supplying the plurality of latent space encodings and the additional features output by the first machine learning network to a second machine learning network that outputs data characterizing the FFR pullback of the coronary blood vessel considering the simulated or planned treatment of the coronary blood vessel when the input data is provided.

[0276] The operation of clause B2 can follow any or some of the operations of clauses A1 - A20 and / or clause B1 above.

[0277] B3. Further including displaying or outputting the data characterizing the FFR pullback of the coronary blood vessel considering the simulated or planned treatment of the coronary blood vessel. The method according to clause B1 or B2.

[0278] B4. Generating first feature data characterizing the presence of zero or more branches or side branches along the axial trajectory of the coronary blood vessel, and supplying the first feature data to the first machine learning system for use in generating the plurality of latent space encodings and the additional feature data. The method according to clause B1 or B2 or B3.

[0279] B5. The first feature data is generated from the analysis of the MPR image; and / or The first feature data is generated from the analysis of the volume image dataset; and / or The first feature data is generated from a coronary artery centerline tree derived from the volume image data set, The method according to item B4.

[0280] B6. The additional features characterized by the additional feature data output by the first machine learning network (and which may be adjusted by the method of B1) include at least one feature related to the lumen characteristics (such as lumen area and / or lumen attenuation) of the cardiovascular concern along the axial trajectory of the cardiovascular concern. The method according to any one of items B1 to B5.

[0281] B7. The additional features characterized by the additional feature data output by the first machine learning network (and which may be adjusted by the method of B1) include at least one feature related to the plaque characteristics (such as calcium plaque area, soft plaque area, mixed plaque area) of the cardiovascular concern along the axial trajectory of the cardiovascular concern. The method according to any one of items B1 to B6.

[0282] B8. Further comprising generating myocardial feature data characterizing a local portion of the myocardium related to the cardiovascular concern and supplying the myocardial feature data as an input to the second machine learning network for use in generating data characterizing the FFR pullback of the cardiovascular concern. The method according to any one of items B1 to B7.

[0283] B9. The first machine learning network comprises a variational autoencoder having an encoder portion that generates the plurality of latent space encodings, and the encoder portion is trained using unsupervised learning. The method according to any one of items B1 to B8.

[0284] B10. The first characteristic data of item B3 is input into the convolutional block of the encoder part, The method according to items B1 to B9.

[0285] B11. The first machine learning network is supplied with a subset of the latent space encoding generated by the encoder part, and is configured to generate the additional characteristic data when the subset of the latent space encoding is given as an input, including at least one auxiliary decoder part, The method according to any one of items B1 to B9.

[0286] B12. The at least one auxiliary decoder part is trained by supervised learning using training data including reference annotations based on the measurement or extraction of corresponding characteristics of a plurality of patients, The method according to any one of items B1 to B9.

[0287] B13. The second machine learning network is trained by supervised learning using training data including reference annotations based on the measurement of FFR pullback values or FFR drop values associated with the vascular centerline points of a plurality of patients, The method according to any one of items B1 to B12.

[0288] B14. The second machine learning network is further configured to output the FFR value of the blood vessel, and the second machine is trained by supervised learning using training data including reference annotations based on the measured values of the FFR values associated with the blood vessels of a plurality of patients, The method according to any one of items B1 to B13.

[0289] B15. The second machine learning network is further configured to output data representing the prediction of the existence of a functionally significant stenosis, and the second machine learning network is trained by supervised learning using training data representing the existence of functionally significant stenosis in a plurality of patients, The method according to any one of clauses B1 to B14.

[0290] B16. The second machine learning network includes a convolutional neural network, and this convolutional neural network is trained by supervised learning using training data including reference annotations for the output data of the second machine learning network. The method according to any one of clauses B1 to B15.

[0291] B17. The reference annotation is derived by manual segmentation and / or automatic segmentation of the corresponding volume image data. The method according to clause B16.

[0292] B18. The convolutional neural network of the second machine learning system includes a regression head that generates an FFR decrease along the axial trajectory of the target blood vessel, and an output stage that generates the FFR pullback output by the second machine learning system. The method according to clause B16 or B17.

[0293] B19. The convolutional neural network of the second machine learning system further includes a first classification head that outputs data representing the FFR value of a blood vessel. The method according to clauses B16 to B18.

[0294] B20. The convolutional neural network of the second machine learning system further includes a second classification head that outputs data representing the prediction of the presence of a functionally significant stenosis. The method according to clauses B16 to B19.

[0295] B21. The relevant cardiovascular vessel includes the coronary artery or coronary artery tree. The method according to any one of clauses B1 to B20.

[0296] B22. The volume image data set comprises CCTA image data. The method according to any one of clauses B1 to B21.

[0297] B23. A system for evaluating the occlusion of a patient's relevant cardiovascular vessel, comprising at least one processor configured to perform any or some of the operations of clauses B1 to B22 when executing program instructions stored in a memory.

[0298] B24. Further comprising an image acquisition subsystem configured to acquire the volume image data set. The system according to clause B23.

[0299] B25. Further comprising a display subsystem configured to display the data characterizing the severity of the anatomical lesion of the relevant cardiovascular vessel. The system according to clause B23.

[0300] B26. A non-transitory program storage device tangibly embodying a program of instructions executable on a machine for performing any or some of the operations of clauses B1 to B22, including simulation or treatment planning of the occlusion of a patient's relevant cardiovascular vessel.

[0301] Clause set C1:

[0302] The embodiments disclosed in this specification can provide a method and system for extracting a coronary artery tree from volumetric image data of a patient's cardiovascular system, which may include one, some, or all of the following operations: Obtaining the volumetric image data set of the cardiovascular system; Tracking a plurality of seed points within the image data set; Using the plurality of seed points to extract an initial representation of the coronary artery tree within the image data set; Inputting the initial representation of the coronary artery tree into a first ensemble of graph convolutional neural networks to generate a refined representation of the coronary artery tree; and Using a second ensemble of graph convolutional neural networks to generate labels for segments of the refined representation of the coronary artery tree.

[0303] C2. The initial and refined representations of the coronary artery tree represent the coronary artery tree as an undirected tree graph, where each point on the centerline of the coronary artery segment corresponds to a node of the tree graph, and the connections between the centerline points are represented by undirected edges of the tree graph. The method according to clause C1.

[0304] C3. The initial representation of the coronary artery tree is constructed by tracking the centerline of the coronary artery from the seed points. The method according to clause C1 or C2.

[0305] C4. The initial representation of the coronary artery tree is derived by predicting the positions of the seed points and the coronary artery inlet using two convolutional neural networks. The method according to clause C1 or C2 or C3.

[0306] C5. Deriving, by using a convolutional neural network configured to predict a direction and a step size from one or more end nodes (nodes having less than two edges) of the graph, adding new nodes to the tree to generate a resulting subgraph, and merging the subgraph if there are duplicates, where the initial representation of the coronary artery tree is The method according to any one of Clauses C1 to C4.

[0307] C6. Deriving, where the initial representation of the coronary artery tree is by grouping adjacent centerline points of the tree graph to create segments The method according to any one of Clauses C1 to C5.

[0308] C7. Characterizing the segments using a set of features selected from the group including the position, orientation, shape, or appearance of the segments The method according to Clause C6.

[0309] C8. The first ensemble of graph convolutional neural networks is configured to perform a binary classification that distinguishes actual coronary artery segments (positive class) from structures such as other blood vessels (negative class) The method according to any one of Clauses C1 to C7.

[0310] C9. The first ensemble of graph convolutional neural networks is configured to adopt a multi - resolution graph ensemble strategy in which predictions of multiple graphs at different resolutions are back - projected onto a fine - grained graph The method according to any one of Clauses C1 to C8.

[0311] C10. The second ensemble of the graph convolutional neural network is trained with graphs of different resolutions for anatomical labeling, The method according to any one of Clauses C1 to C9.

[0312] C11. Further including an operation of displaying or outputting the refined representation of the coronary artery tree and / or the label of the segment of the coronary artery tree. The method according to any one of Clauses C1 to C10.

[0313] C12. A system for extracting a coronary artery tree from volumetric image data of a patient's cardiovascular system, comprising at least one processor configured to perform any one or some of the operations of Clauses C1 to C12 when executing program instructions stored in a memory.

[0314] C13. Further comprising an image acquisition subsystem configured to acquire the volumetric image data set. The system according to Clause C13.

[0315] C14. A display subsystem configured to display any one of Clauses C1 to C10, further comprising an operation of displaying or outputting a refined representation of a coronary artery tree and / or a label of a segment of the coronary artery tree. The system according to Clause C13.

[0316] C15. A non-transitory program storage device tangibly embodying a program of instructions executable on a machine for performing any one or some of the operations of Clauses C1 to C11 to extract a coronary artery tree from volumetric image data.

[0317] All the features appearing in the above set of clauses can be combined among themselves and with any of the features appearing in the appended claims.

Claims

1. A method for evaluating the occlusion of a patient's relevant cardiovascular vessels, comprising: obtaining a volume image dataset of the relevant cardiovascular vessels; analyzing the volume image dataset to extract data representing the axial trajectory of the relevant cardiovascular vessels; generating a multi-planar reformation (MPR) image based on the volume image dataset and the data representing the axial trajectory of the relevant cardiovascular vessels; feeding the MPR image as an input to a first machine learning network that outputs feature data characterizing a plurality of features of the relevant cardiovascular vessels along the axial trajectory of the relevant cardiovascular vessels when the MPR image is provided; generating additional data characterizing at least one additional feature of the relevant cardiovascular vessels along the axial trajectory of the relevant cardiovascular vessels by a different analysis separate from the first machine learning network; and feeding, as input data, the data output by the first machine learning network and the additional data to a second machine learning network that outputs data characterizing the severity of an anatomical lesion of the relevant cardiovascular vessels when the input data is provided A method comprising the above steps.

2. The method according to claim 1, further comprising the step of displaying or outputting the data characterizing the severity of the anatomical lesion of the relevant cardiovascular vessels.

3. The additional data is generated from the analysis of the MPR image; and / or The additional data is generated from the analysis of the volume image dataset; and / or The additional data is generated from a coronary artery centerline tree obtained from the volume image dataset, The method according to claim 1 or 2.

4. The method according to any one of claims 1 to 3, wherein the additional data characterizes at least one of the collateral branches and bifurcations along the axial trajectory of the relevant cardiovascular vessels.

5. The method according to any one of claims 1 to 4, wherein the additional data characterizes at least one of the soft plaque area, mixed plaque area or other specific features along the axial trajectory of the relevant cardiovascular vessels.

6. The method according to any one of claims 1 to 5, wherein the additional data further characterizes a local portion of the myocardium associated with the relevant cardiovascular vessels.

7. The data output by the second machine learning network includes the fractional flow reserve (FFR) value of the entire coronary blood vessel concerned, and the second machine learning network is trained by supervised learning using training data including reference annotations based on measurements of FFR values of a plurality of patients, The method according to any one of claims 1 to 6.

8. The data output by the second machine learning network includes the fractional flow reserve (FFR) value of the centerline points along the coronary blood vessel concerned, and the second machine learning network is trained by supervised learning using training data including reference annotations based on measurements of FFR values associated with the vascular centerline points of a plurality of patients, The method according to any one of claims 1 to 7.

9. The data output by the second machine learning network represents a prediction of the presence of a functionally significant stenosis, and the second machine learning network is trained by supervised learning using training data including reference annotations representing the presence of a functionally significant stenosis in a plurality of patients, The method according to any one of claims 1 to 8.

10. The plurality of features characterized by the feature data output by the first machine learning network includes at least one feature related to the lumen characteristics (such as lumen area and / or lumen attenuation) of the coronary blood vessel along the axial trajectory of the coronary blood vessel The method according to any one of claims 1 to 9.

11. The plurality of features characterized by the feature data output by the first machine learning network includes at least one feature related to the plaque characteristics (such as calcium plaque area, soft plaque area, mixed plaque area) of the coronary blood vessel along the axial trajectory of the coronary blood vessel. The method according to any one of claims 1 to 10.

12. The first machine learning network comprises a convolutional neural network trained using training data including reference annotations of the plurality of features characterized by the feature data output by the first machine learning network The method according to any one of claims 1 to 11.

13. wherein the reference annotation is derived by manual segmentation of the corresponding volume image data and / or automatic segmentation of the corresponding volume image data, The method according to claim 12. **Claim 14** The method according to any one of claims 1 to 13, comprising a convolutional neural network trained using training data including the volume image data and corresponding reference annotations characterizing the severity of the anatomical lesion of the relevant cardiovascular system. **Claim 15** wherein the reference annotation is derived by manual segmentation of the corresponding volume image data and / or automatic segmentation of the corresponding volume image data, The method according to claim 14. **Claim 16** The method according to claim 14, wherein the convolutional neural network of the second machine learning system includes a regression head that outputs a fractional flow reserve (FFR) value. The method according to claim 14. **Claim 17** The method according to claim 16, wherein the convolutional neural network of the second machine learning system further includes an accumulator that outputs a fractional flow reserve (FFR) value for a centerline point along the relevant cardiovascular system. The method according to claim 16. **Claim 18** The method according to any one of claims 14 to 17, wherein the convolutional neural network of the second machine learning system includes a classification head that outputs data representing the prediction of the presence of a functionally significant stenosis. The method according to any one of claims 14 to 17. **Claim 19** The method according to any one of claims 1 to 18, wherein the relevant cardiovascular system comprises a coronary artery or a coronary artery tree. **Claim 20** The method according to any one of claims 1 to 19, wherein the volume image data set comprises CCTA image data. The method according to any one of claims 1 to 19. **Claim 21** A system for evaluating the occlusion of a patient's relevant cardiovascular system, comprising at least one processor configured to execute, when executing program instructions stored in a memory, some or all of the operations of any one of claims 1 to 20. **Claim 22** The system according to claim 21, further comprising an image acquisition subsystem configured to acquire the volume image data set. **Claim 23** The system of claim 22, further comprising a display subsystem configured to display the data characterizing the severity of the cardiovascular anatomical lesion. **Claim 24** A non-transitory program storage device tangibly embodying a program of instructions executable on a machine to perform some or all of the operations of any one of claims 1 to 20 for evaluating an occlusion of a patient's cardiovascular system.

Images