Coronary lumen and reference wall segmentation for automated assessment of coronary artery disease

Abstract

Systems and methods for automated assessment of a vessel are provided. One or more input medical images of a vessel of a patient are received. A plurality of vessel assessment tasks for assessing the vessel are performed using a machine learning based model trained by utilizing multi-task learning. The plurality of vessel assessment tasks includes segmentation of a reference wall of the vessel from the one or more input medical images and segmentation of a lumen of the vessel from the one or more input medical images. Results of the plurality of vessel assessment tasks are output.

Description

Technical Field

[0001] This invention relates generally to the automated assessment of coronary artery disease (CAD), and particularly to the segmentation of the coronary artery lumen and reference wall for the automated assessment of coronary artery disease. Background Technology

[0002] Coronary artery disease (CAD) is characterized by the buildup of plaque in the coronary arteries, leading to narrowing of the arteries and reduced blood flow to the heart. This narrowing of the coronary arteries is called stenosis. Stenosis assessment is performed based on coronary CTA (computed tomography angiography) imaging.

[0003] Various methods have been proposed for the automated assessment of stenosis. One conventional method estimates the diameter or area of ​​the lumen between the healthy ends of the coronary arteries based on coronary CTA imaging, either by interpolating the diameter or area of ​​the lumen between the healthy ends or by iteratively fitting a line to a diameter or area profile of the lumen. However, this conventional method fails to accurately assess stenosis when plaque buildup is extensive and long, or when plaque buildup passes through bifurcations in the coronary arteries. Another conventional method employs an AI-based system trained to assess stenosis based on a synthetically generated vessel tree. However, this conventional method also suffers from the limitation of using only a diameter or area profile of the lumen, thus reducing its accuracy. Summary of the Invention

[0004] According to one or more embodiments, a system and method for automated assessment of blood vessels are provided. One or more input medical images of a patient's blood vessels are received. Multiple blood vessel assessment tasks for assessing the blood vessels are performed using a machine learning-based model trained by leveraging multi-task learning based on shared features extracted from the one or more input medical images. The multiple blood vessel assessment tasks include segmentation of a reference wall of the blood vessel from the one or more input medical images and segmentation of the lumen of the blood vessel from the one or more input medical images. The results of the multiple blood vessel assessment tasks are output.

[0005] In one embodiment, the machine learning-based model is trained for segmentation of the reference wall of the blood vessel based on regularization of anatomical tapering.

[0006] In one embodiment, the machine learning-based model is trained for segmenting both the reference wall and lumen of the blood vessels based on regularization of the consistency between the segmentation results of the reference wall and lumen segments of the blood vessels in regions without anomalies or lesions. The regularization of the consistency between the segmentation results of the reference wall and lumen segments of the blood vessels in regions without anomalies or lesions is based on ground truth labels for the anomalies and ground truth labels for the lesions.

[0007] In one embodiment, the machine learning-based model is trained for segmenting reference walls of blood vessels based on regularization of the segmentation of reference walls in regions with anomalies or lesions. The regularization of the segmentation of reference walls in regions with anomalies or lesions is based on ground truth labels for the anomalies and ground truth labels for the lesions.

[0008] In one embodiment, the machine learning-based model is trained for stenosis grading based on regularization of consistency between ground truth stenosis grading and stenosis grading based on vascular reference wall segmentation results and lumen segmentation results.

[0009] In one embodiment, the machine learning-based model is trained to be used for segmenting the lumen of blood vessels based on the consistency between ground truth lumen segmentation and lumen segmentation results.

[0010] In one embodiment, the plurality of vascular assessment tasks further include image-based stenosis grading of stenosis in the vascular system, and wherein the results of image-based stenosis grading, segmentation of the reference wall, and segmentation of the lumen are consistent.

[0011] In one embodiment, the degree of stenosis in the blood vessel is determined based on the segmentation of the reference wall of the blood vessel and the segmentation of the lumen of the blood vessel.

[0012] In one embodiment, an uncertainty estimate is determined for each of the plurality of vascular assessment tasks.

[0013] These and other advantages of the present invention will become apparent to those skilled in the art from the following detailed description and accompanying drawings. Attached Figure Description

[0014] Figure 1 A method for automated assessment of blood vessels according to one or more embodiments is shown;

[0015] Figure 2A multi-task learning framework for a machine learning-based model according to one or more embodiments is shown, the machine learning-based model being trained using multi-task learning to perform multiple vascular assessment tasks;

[0016] Figure 3 The results of a vascular assessment task, which generates segmentation of the reference wall for a vascular vessel and segmentation of the lumen for a vascular vessel according to one or more embodiments, are shown.

[0017] Figure 4 Results of a vascular assessment task for vascular reference wall segmentation and vascular lumen segmentation from various input medical images, according to one or more embodiments, are shown.

[0018] Figure 5 A framework for training a multi-task AI (artificial intelligence) system to perform multiple vascular assessment tasks, according to one or more embodiments, is shown.

[0019] Figure 6 A multi-task learning framework for a machine learning-based model determined using uncertainty estimation according to one or more embodiments is shown, the machine learning-based model being trained using multi-task learning to perform multiple vascular assessment tasks.

[0020] Figure 7 An exemplary artificial neural network that can be used to implement one or more embodiments is shown;

[0021] Figure 8 A convolutional neural network that can be used to implement one or more embodiments is shown; and

[0022] Figure 9 A high-level block diagram of a computer that can be used to implement one or more embodiments is shown. Detailed Implementation

[0023] This invention generally relates to methods and systems for segmenting the coronary artery lumen and reference wall for automated assessment in CAD (coronary artery disease). Embodiments of the invention are described herein to provide a visual understanding of the methods and systems. Digital images typically consist of digital representations of one or more objects (or shapes). The digital representations of objects are generally described herein in terms of identifying and manipulating them. Such manipulation is a virtual manipulation performed in the memory or other circuitry / hardware of a computer system. Therefore, it is to be understood that embodiments of the invention can be performed within a computer system using data stored within the computer system.

[0024] The embodiments described herein provide a multi-task learning framework for end-to-end training of a single machine learning-based model to perform reference wall segmentation and lumen segmentation of a blood vessel, along with other vascular assessment tasks for CAD evaluation (e.g., stenosis grading). The joint determination of reference wall and lumen segmentation performed by this single machine learning-based model provides accurate segmentation and robust stenosis grading results against image artifacts and poor image quality. By utilizing this single machine learning-based model, consistency in results across the multiple vascular assessment tasks is ensured. Furthermore, this single end-to-end machine learning-based model can produce meaningful results regardless of the failure of individual medical imaging analysis tasks.

[0025] Figure 1 A method 100 for automated assessment of blood vessels according to one or more embodiments is illustrated. The steps of method 100 can be performed by one or more suitable computing devices (such as, for example...) Figure 9 The computer (902) is used to execute this.

[0026] exist Figure 1 At step 102, one or more input medical images of the patient's blood vessels are received. The patient's blood vessels can be the patient's arteries, veins, or any other blood vessels. For example, a blood vessel can be a branch of the patient's coronary artery, an intracranial or extracranial blood vessel, the patient's aorta, a peripheral blood vessel, etc. The input medical images can depict plaques or other abnormalities (e.g., lesions, nodules, or any other abnormalities) on the blood vessel walls.

[0027] In one embodiment, the input medical image comprises a cross-sectional image of the blood vessel sampled along the vessel. However, the input medical image may include any suitable image of the blood vessel, and is not limited to a cross-sectional image of the blood vessel. In some embodiments, the input medical image may include features such as, for example, a reformatted view of the blood vessel, geometric features of the blood vessel (e.g., distances to anatomical landmarks or other anatomically interested objects, estimates of the blood vessel diameter, etc.), anatomical features of the blood vessel (e.g., labels identifying the blood vessel, indicators of bifurcations in the blood vessel, etc.), or any other suitable features of the blood vessel.

[0028] In one embodiment, the input medical image is a CT (computed tomography) image, such as, for example, a CTA (computed tomography angiography) image. This CTA image can be a monophasic CTA image or a multiphasic CTA image. However, the input medical image can belong to any other suitable modality, such as, for example, MRI (magnetic resonance imaging), X-ray, US (ultrasound), or any other modality or combination of modalities. The input medical image can include a 2D (two-dimensional) image or a 3D (three-dimensional) volume and can include a single image or multiple images (e.g., a sequence of images acquired over time). The input medical image can be acquired directly from an image acquisition device such as, for example, a CT scanner (e.g., ...). Figure 9 The image acquisition device 914 is received, or it can be received by loading previously acquired images from the storage device or memory of the computer system or by receiving images from a remote computer system.

[0029] exist Figure 1 At step 104, a machine learning-based model is used to perform multiple vascular assessment tasks for evaluating blood vessels. This machine learning-based model is trained using multi-task learning based on shared features extracted from the one or more input medical images. These multiple vascular assessment tasks can be any suitable task for evaluating blood vessels.

[0030] In one embodiment, the multiple vascular assessment tasks include the segmentation of a reference wall of the blood vessel. As used herein, a reference wall of a blood vessel refers to the wall of the blood vessel when it is healthy, excluding any disease or abnormality (e.g., plaque, lesion, nodule, or any other abnormality) on the wall of the blood vessel. Various metrics, such as, for example, the diameter (e.g., effective diameter or minimum / maximum diameter) or area of ​​the reference wall of the blood vessel, can be calculated based on the segmented reference wall of the blood vessel.

[0031] In one embodiment, the multiple vascular assessment tasks include the segmentation of the lumen of the vascular vessels. Various metrics, such as, for example, the diameter (e.g., effective diameter or minimum / maximum diameter) or area of ​​the lumen of the vascular vessels, can be calculated based on the segmented lumen of the vascular vessels.

[0032] In one embodiment, the plurality of vascular assessment tasks includes image-based stenosis grading of stenosis in blood vessels. As used herein, image-based stenosis grading refers to stenosis grading performed directly from shared features without using segmentation results of the blood vessel lumen. Stenosis can be graded or classified as, for example, normal, minimal, mild, moderate, severe, or occluded, or can be graded as percentage stenosis.

[0033] These multiple vascular assessment tasks may include any other vascular assessment tasks, such as the detection and classification of diseases (e.g., coronary artery disease), the detection and classification of artifacts in input medical images, the detection and classification of abnormalities in input medical images, the detection and classification of abnormalities (e.g., myocardial bridging, or abnormalities due to previous interventions (e.g., stents, bypass grafts, etc.), and the image-based determination of vascular hemodynamic parameters (e.g., FFR (fractional flow reserve), CFR (coronary flow reserve), iFR (instantaneous wave ratio), etc.).

[0034] In one embodiment, the machine learning-based model includes: 1) an encoder for encoding the one or more input medical images into shared features (i.e., latent features or latent representations); and 2) a plurality of decoders, each decoder for decoding the shared features to perform a corresponding one of the plurality of vascular assessment tasks. The machine learning-based model can be any suitable machine learning-based model for performing the plurality of vascular assessment tasks, such as, for example, a DNN (Deep Neural Network), a CNN (Convolutional Neural Network), DI2IN (Depth Image-to-Image Network), etc. In one embodiment, the machine learning-based model is based on the detailed description below. Figure 2 Frame 200 or Figure 6 The framework 600 is used to implement this. Multi-task learning is used to train the machine learning-based model on labeled training data during a previous offline or training phase to perform the multiple vascular assessment tasks. In one embodiment, the machine learning-based model can be implemented according to the following detailed description. Figure 5 The framework 500 is used for training. Once trained, the trained machine learning-based model is applied (e.g., at step 104) to perform the multiple vascular assessment tasks during the online or testing phase.

[0035] Figure 2 A multi-task learning framework 200 based on a machine learning model is illustrated according to one or more embodiments. This machine learning model is trained using multi-task learning to perform multiple vascular assessment tasks. Framework 200 may be in... Figure 1The framework of the machine learning-based model applied at step 104. In framework 200, the encoder 204 of the machine learning-based model receives one or more input medical images 202 of the patient's blood vessels as input and encodes the one or more input medical images 202 into shared features 206. The encoder 204 may be a VAE (variational autoencoder). The decoders D1 208-A, D2 208-B, D3 208-C, and Dn 208-N (collectively referred to as multiple decoders 208) of the machine learning-based model decode the shared features 206 to perform corresponding blood vessel assessment tasks 210-A, 210-B, 210-C, 210-D, and 210-N (collectively referred to as multiple blood vessel assessment tasks 210). As shown in frame 200, vascular assessment task 210-A is used for reference wall segmentation, vascular assessment task 210-B for lumen segmentation, vascular assessment task 210-C for image-based stenosis grading (e.g., as normal, minimal, mild, moderate, severe, or occluded), and vascular assessment task 210-N for other vascular assessment tasks, such as lesion detection, artifact detection, plaque component classification, etc. Since these multiple vascular assessment tasks 210 are jointly performed by a single machine learning-based model from shared features 206, consistency in the results of each of the multiple vascular assessment tasks 210 is ensured. Additionally, the machine learning-based model can produce meaningful results regardless of failures in individual tasks.

[0036] exist Figure 1 At step 106, the results of the multiple vascular assessment tasks are output. For example, the results of the multiple vascular assessment tasks can be output in the following ways: displaying the results of the multiple vascular assessment tasks on a display device of the computer system, storing the results of the multiple vascular assessment tasks in the memory or storage device of the computer system, or transmitting the results of the multiple vascular assessment tasks to a remote computer system.

[0037] In one embodiment, the results of the multiple vascular assessment tasks can be output to other systems. In another embodiment, the results of the multiple vascular assessment tasks can be output to a fully automated coronary artery analysis system or a fully automated FFR (fractional flow reserve) prediction system. Such an FFR prediction system will be robust to image quality issues (e.g., artificial narrowing of segmentation results due to imaging artifacts). In yet another embodiment, the results of the multiple vascular assessment tasks can be output as soft or hard regularization constraints to an outer wall segmentation algorithm to improve results for complex lesions (e.g., non-calcified plaques).

[0038] In one embodiment, the segmentation results of the reference wall of the blood vessel and the segmentation results of the lumen of the blood vessel can be presented as an overlay on the input medical image.

[0039] In one embodiment, the degree of stenosis in a blood vessel can be determined based on the segmentation of the reference wall of the blood vessel and the segmentation of the lumen of the blood vessel.

[0040] Figure 3 The results 300 of a vascular assessment task, generated according to one or more embodiments, show the segmentation of the reference wall and the lumen of the vascular vessel. Image 302 shows the segmentation of the reference wall of the vascular vessel superimposed on an MPR (multiplanar reformation) image, and image 304 shows the segmentation of the lumen of the vascular vessel superimposed on an MPR image. The segmentation of the reference wall and the segmentation of the lumen can be... Figure 2 The corresponding results of vascular assessment tasks 210-A and 210-B are shown. Graph 306 compares the effective diameter of the reference wall (line 308) with the effective diameter of the lumen (line 310). As shown in graph 306, the diameter of the lumen of the vessel deviates significantly from the diameter of the reference wall of the vessel at point 312, indicating the narrowing 314 shown in the lumen segmentation results in image 304.

[0041] Figure 4 The results 400 of a vascular assessment task, including vascular reference wall segmentation and vascular lumen segmentation from various input medical images according to one or more embodiments, are shown. The segmentation of the reference wall and the lumen can be... Figure 2 The corresponding results for vascular assessment tasks 210-A and 210-B are shown. Images 402-A, 402-B, and 402-C show the segmentation of the reference wall of the corresponding blood vessel overlaid on the MPR image. Images 404-A, 404-B, and 404-C show the segmentation of the lumen of the corresponding blood vessel overlaid on the MPR image. Plots 406-A, 406-B, and 406-C compare the effective diameter of the reference wall (lines 408-A, 408-B, and 408-C) with the effective diameter of the lumen (410-A, 410-B, and 410-C). As shown in plot 406-A, the diameter of the lumen of the blood vessel deviates significantly from the diameter of the reference wall of the blood vessel at point 412-A, indicating the narrowing 414-A shown in the lumen segmentation results in image 404-A. As shown in curve 406-B, the diameter of the vessel lumen deviates significantly from the diameter of the reference wall of the vessel at point 412-B, indicating the narrowing at 414-B shown in the segmentation results of the lumen in image 404-B. As shown in curve 406-C, the diameter of the vessel lumen deviates significantly from the diameter of the reference wall of the vessel at point 412-C, indicating the narrowing at 414-C shown in the segmentation results of the lumen in image 404-C.

[0042] Advantageously, the embodiments described herein enable fully automated assessment of coronary artery disease while producing interpretable results for vascular assessment tasks with discovered localization. The embodiments described herein jointly perform reference wall segmentation and lumen segmentation of the vessels, thereby increasing accuracy and robustness to image artifacts and poor image quality. Furthermore, the embodiments described herein jointly train a single end-to-end machine learning-based model for performing multiple vascular assessment tasks. This ensures consistency of results across different vascular assessment tasks, as well as improved performance and generalization through feature sharing among related vascular assessment tasks. The embodiments described herein can be extended to coronary vascular analysis of invasively acquired 2D angiography images.

[0043] Figure 5 A framework 500 for training a multi-task AI (artificial intelligence) system 504 to perform multiple vascular assessment tasks is illustrated according to one or more embodiments. In one embodiment, the multi-task AI system 504 may be in... Figure 1 The machine learning-based model applied in step 104 Figure 2 The machine learning-based model shown, or Figure 6 The machine learning-based model shown is described below. Framework 500 uses multi-task learning to train a multi-task AI system 504 to perform multiple vascular assessment tasks. As shown in framework 500, the multi-task AI system 504 receives coronary CTA training images 502 as input and generates predictions for multiple vascular assessment tasks, including lumen segmentation 506, reference wall segmentation 508, and image-based stenosis grading 510. These predictions, along with ground truth data, are used to optimize an objective function for training the multi-task AI system 504. This objective function is computed as a weighted sum of different loss terms 512-522 with supervised, weakly supervised, regularized, and / or weakly supervised regularization.

[0044] The multi-task AI system 504 is trained by optimizing an objective function, which includes one or more of the following loss terms:

[0045] 1) Full supervision 512 is used to enforce consistency between ground truth lumen 524 and lumen segmentation;

[0046] 2) Regularization 514, used to enforce regularization based on coronary anatomy tapering;

[0047] 3) Regularization with weak supervision 516 to enforce consistency between lumen segmentation and reference wall segmentation in areas without abnormalities or lesions (using ground truth markers 528 for abnormalities (e.g., artifacts, myocardial bridging and stents) and ground truth markers and stenosis grades 530 for lesions (e.g., plaques).

[0048] 4) Regularization with weak supervision 518 is used to enforce reference wall segmentation coverage of the lumen in abnormal or diseased areas (using ground truth markers 528 and ground truth markers and stenosis levels 530) to ensure complete coverage of the lumen segmentation output within the reference wall segmentation output;

[0049] 5) Weak supervision 520, used to enforce consistency between the ground truth stenosis classification 530 and the segmentation-based stenosis classification, which is based on the segmentation results of the reference wall of the blood vessel and the segmentation results of the lumen of the blood vessel; and

[0050] 6) Weak supervision 522, used to enforce consistency between ground truth narrowing 530 and image-based point-by-point narrowing 530.

[0051] like Figure 5 As shown, the multi-task AI system 504 is trained for lumen segmentation 506 using full supervision 512, regularization with weak supervision 516, and weak supervision 520. The multi-task AI system 504 is trained for reference wall segmentation 508 using regularization 514, regularization with weak supervision 516, regularization with weak supervision 518, and weak supervision 520. The multi-task AI system 504 is trained for image-based stenosis grading 510 using weak supervision 522. It is important to note that the multi-task AI system 504 is trained for reference wall segmentation 508 without using ground truth markers or annotations of the reference wall, because such ground truth markers are unavailable for vessels with plaques and other lesions on their walls. Exemplary loss functions may be, for example (but not limited to), dice or cross-entropy loss for fully supervised 512, sum of positive differences of consecutive terms of effective diameter derived from reference wall segmentation for regularization 514, dice or cross-entropy loss for regularizations 516 and 518 with weak supervision, cross-entropy classification loss for weak supervision 520, and L1 / L2 loss for weak supervision 520.

[0052] In one embodiment, the multi-task AI system 504 may be trained using invasively measured ground truth FFR (fractional flow reserve) values ​​or ground truth FFR values ​​calculated based on CT images (CT-FFR) for weakly supervised optimization to generate results conforming to each other for vascular assessment tasks (e.g., segmentation or image-based FFR calculation). For example, the multi-task AI system 504 may be trained to generate segmentation results that conform to image-based FFR prediction, or invasive FFR measurement, or CT-based FFR measurement.

[0053] In one embodiment, in Figure 1 The machine learning-based model applied at step 104 can determine a confidence metric for the results of the multiple vascular assessment tasks. The confidence metric can be a holistic confidence metric for the multiple vascular assessment tasks, indicating the level of consistency among them. Alternatively, the confidence metric can be a confidence metric determined for each result of the multiple vascular assessment tasks. The confidence metric can be represented in any suitable form, such as, for example, a confidence score, a heatmap representing confidence, etc. In one example, the confidence metric is determined for the reference wall segmentation and the lumen segmentation of the vascular segment. In one embodiment, the confidence metric can be determined by varying the threshold used to generate the contour / mesh from the segmentation. In another embodiment, the confidence metric can be determined as an uncertainty estimate that can be predicted in conjunction with the output segmentation probability. For example, the last layer of the machine learning-based model can be replaced with a Gaussian process to output a probability distribution of the uncertainty estimate, such as... Figure 6 As shown in the image.

[0054] Figure 6 A multi-task learning framework 600 is illustrated, which utilizes uncertainty estimation to determine a machine learning-based model according to one or more embodiments. This machine learning-based model is trained using multi-task learning to perform multiple vascular assessment tasks. Framework 600 may be in... Figure 1The framework of the machine learning-based model applied at step 104. In framework 600, the encoder 604 of the machine learning-based model receives one or more input medical images 602 of the patient's blood vessels as input and encodes the one or more input medical images 602 into shared features 606. The decoders Dr 608-A, D1 608-B, D2 608-C, D3 608-D, and Dn 608-N (collectively referred to as multiple decoders 608) of the machine learning-based model decode the shared features 606 to perform corresponding blood vessel assessment tasks 612-A, 612-B, 612-C, 612-D, and 612-N (collectively referred to as multiple blood vessel assessment tasks 612). As shown in frame 600, vascular assessment task 612-A is used for reconstruction of the input medical image 602, vascular assessment task 612-B is used for reference wall segmentation, vascular assessment task 612-C is used for lumen segmentation, vascular assessment task 612-D is used for image-based stenosis grading (normal, minimal, mild, moderate, severe, or occlusive), and vascular assessment task 612-N is used for other vascular assessment tasks, such as lesion detection, artifact detection, and plaque component classification. The last layer of decoders 608-B, 608-C, 608-D, and 608-N is replaced with corresponding Gaussian processes 610-B, 610-C, 610-D, and 610-N to output a probability distribution of uncertainty estimates. Gaussian processes are not applied to decoder 608-A for reconstruction task 612-A of the input medical image 602. Reconstruction task 612-A regularizes the manifold used for training, thereby regularizing shared features 606.

[0055] The embodiments described herein relate to the claimed system and the claimed method. Features, advantages, or alternative embodiments described herein may be assigned to other claims, and vice versa. In other words, the claims for the system can be improved using features described or claimed in the context of the method. In this case, the functional characteristics of the method are embodied by providing the target unit of the system.

[0056] Furthermore, certain embodiments described herein relate to methods and systems for utilizing trained machine learning-based networks (or models), and to methods and systems for training machine learning-based networks. Features, advantages, or alternative embodiments described herein may be assigned to other claims, and vice versa. In other words, the claims for methods and systems for training machine learning-based networks may be improved using features described or claimed in the context of methods and systems for utilizing trained machine learning-based networks, and vice versa.

[0057] Specifically, the trained machine learning-based network used in the embodiments described herein can be adapted by methods and systems for training machine learning-based networks. Furthermore, the input data of the trained machine learning-based network can include advantageous features and embodiments of the training input data, and vice versa. Similarly, the output data of the trained machine learning network can include advantageous features and embodiments of the output training data, and vice versa.

[0058] Generally speaking, trained machine learning-based networks mimic human cognitive functions that connect with other human minds. In particular, through training on training data, trained machine learning-based networks can adapt to new situations and detect and extrapolate patterns.

[0059] Generally, the parameters of a machine learning-based network can be adapted through training. Specifically, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of a trained machine learning-based network can be iteratively adapted through several training steps.

[0060] Specifically, the trained machine learning-based network can include neural networks, support vector machines, decision trees, and / or Bayesian networks, and / or the trained machine learning-based network can be based on k-means clustering, Q-learning, genetic algorithms, and / or association rules. Specifically, the neural network can be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, the neural network can be an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0061] Figure 7 An embodiment of an artificial neural network 700 according to one or more embodiments is shown. Alternative terms for "artificial neural network" are "neural network," "artificial neural network," or "neural network." The artificial neural network 700 can be used to implement the machine learning networks described herein, including... Figure 1 The machine learning-based model applied in step 104 Figure 2 The encoder 204 and multiple decoders 208, Figure 5 Multi-task AI system 504 and Figure 6 The encoder 604 and multiple decoders 608.

[0062] The artificial neural network 700 includes nodes 702-722 and edges 732, 734...736, where each edge 732, 734...736 is a directed connection from a first node 702-722 to a second node 702-722. Generally, the first node 702-722 and the second node 702-722 are different nodes 702-722, but it is also possible that the first node 702-722 and the second node 702-722 are the same. For example, in... Figure 7 In the diagram, edge 732 is a directed connection from node 702 to node 706, and edge 734 is a directed connection from node 704 to node 706. Edges 732, 734...736 from the first node 702-722 to the second node 702-722 are also represented as "incoming edges" for the second node 702-722 and "outgoing edges" for the first node 702-722.

[0063] In this embodiment, nodes 702-722 of the artificial neural network 700 can be arranged in layers 724-730, wherein these layers may include an inherent order introduced by edges 732, 734...736 between nodes 702-722. Specifically, edges 732, 734...736 may exist only between adjacent layers of nodes. Figure 7 In the illustrated embodiment, there is an input layer 724 comprising only nodes 702 and 704 without incoming edges, an output layer 730 comprising only node 722 without outgoing edges, and hidden layers 726 and 728 between the input layer 724 and the output layer 730. Generally, the number of hidden layers 726 and 728 can be chosen arbitrarily. The number of nodes 702 and 704 in the input layer 724 is typically related to the number of input values ​​of the neural network 700, and the number of nodes 722 in the output layer 730 is typically related to the number of output values ​​of the neural network 700.

[0064] Specifically, (real) numbers can be assigned as values ​​to each node 702-722 of the neural network 700. Here, x (n) i This represents the value of the i-th node 702-722 in the n-th layer 724-730. The values ​​of nodes 702-722 in the input layer 724 are equivalent to the input values ​​of the neural network 700, and the value of node 722 in the output layer 730 is equivalent to the output value of the neural network 700. Furthermore, each edge 732, 734...736 may include a weight as a real number, specifically a real number in the interval [-1, 1] or the interval [0, 1]. Here, w... (m,n) i,jThis represents the weight of the edge between the i-th node 702-722 in layer m (724-730) and the j-th node 702-722 in layer n (724-730). Additionally, the abbreviation w... (n) i,j Defined for weight w (n,n+1) i,j .

[0065] Specifically, in order to calculate the output value of neural network 700, the input value is propagated through the neural network. Specifically, the value of node 702-722 in layer (n+1) 724-730 can be calculated based on the value of node 702-722 in layer n 724-730 using the following formula:

[0066] .

[0067] In this paper, the function f is the transfer function (another term is "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logistic functions, generalized logistic functions, hyperbolic tangent functions, arctangent functions, error functions, smoothstep functions), or rectifier functions. Transfer functions are primarily used for normalization purposes.

[0068] Specifically, these values ​​are propagated layer by layer through the neural network, where the value of the input layer 724 is given by the input of the neural network 700, the value of the first hidden layer 726 can be calculated based on the value of the input layer 724 of the neural network, the value of the second hidden layer 728 can be calculated based on the value of the first hidden layer 726, and so on.

[0069] To set the value w of the edge (m,n) i,j Training data must be used to train the neural network 700. Specifically, the training data includes training input data and training output data (denoted as t). i For the training step, the neural network 700 is applied to the training input data to generate the computed output data. Specifically, the training data and the computed output data include a number of values ​​equal to the number of nodes in the output layer.

[0070] Specifically, the weights within the neural network are recursively adapted using a comparison between the calculated output data and the training data (backpropagation algorithm). Specifically, the weights are changed according to the following formula:

[0071]

[0072] Where γ is the learning rate, and if the (n+1)th layer is not the output layer, it can be based on δ. (n+1)j Recursively apply the number δ (n) j The calculation is as follows:

[0073] ,

[0074] And if the (n+1)th layer is the output layer 730, then it is calculated as follows:

[0075]

[0076] Where f' is the first derivative of the activation function, and y (n+1) j It is the comparison training value of the j-th node of the output layer 730.

[0077] Figure 8 A convolutional neural network 800 according to one or more embodiments is illustrated. The convolutional neural network 800 can be used to implement the machine learning networks described herein, including... Figure 1 The machine learning-based model applied in step 104 Figure 2 The encoder 204 and multiple decoders 208, Figure 5 Multi-task AI system 504 and Figure 6 The encoder 604 and multiple decoders 608.

[0078] exist Figure 8 In the illustrated embodiment, the convolutional neural network 800 includes an input layer 802, a convolutional layer 804, a pooling layer 806, a fully connected layer 808, and an output layer 810. Alternatively, the convolutional neural network 800 may include several convolutional layers 804, several pooling layers 806, several fully connected layers 808, and other types of layers. The order of these layers can be arbitrarily chosen; typically, the fully connected layer 808 is used as the last layer before the output layer 810.

[0079] Specifically, within a convolutional neural network 800, the nodes 812-820 of a layer 802-810 can be considered as arranged as a d-dimensional matrix or a d-dimensional image. In particular, in the two-dimensional case, the value of node 812-820 indexed by i and j in the nth layer 802-810 can be represented as x. (n) [i, j] However, the arrangement of nodes 812-820 in a layer 802-810 has no effect on the computation itself performed within the convolutional neural network 800, because they are given only by the structure and weights of the edges.

[0080] Specifically, the convolutional layer 804 is characterized by forming the structure and weights of the input edges for the convolution operation based on a certain number of kernels. In particular, the structure and weights of the input edges are chosen such that the value x of node 814 of the convolutional layer 804...(n) k Based on the value x of node 812 of the previous layer 802. (n-1) And calculated as convolution x (n) k =K k * x (n-1) In the two-dimensional case, convolution* is defined as:

[0081] .

[0082] Here, the k-th core K k This is a d-dimensional matrix (a two-dimensional matrix in this embodiment), which is typically small compared to the number of nodes 812-818 (e.g., a 3×3 or 5×5 matrix). Specifically, this means that the weights of the incoming edges are not independent but are chosen such that they produce the convolution formula. Specifically, for the kernel, which is a 3×3 matrix, there are only 9 independent weights (each entry in the kernel matrix corresponds to one independent weight), regardless of the number of nodes 812-820 in the corresponding layers 802-810. Specifically, for convolutional layer 804, the number of nodes 814 in this convolutional layer is equal to the number of nodes 812 in the previous layer 802 multiplied by the number of kernels.

[0083] If the nodes 812 of the previous layer 802 are arranged as a d-dimensional matrix, then using multiple kernels can be interpreted as adding another dimension (represented as the "depth" dimension) so that the nodes 814 of the convolutional layer 804 are arranged as a (d+1)-dimensional matrix. If the nodes 812 of the previous layer 802 have already been arranged as a (d+1)-dimensional matrix including the depth dimension, then using multiple kernels can be interpreted as extending along the depth dimension so that the nodes 814 of the convolutional layer 804 are also arranged as a (d+1)-dimensional matrix, where the size of this (d+1)-dimensional matrix with respect to the depth dimension is a multiple of the kernel size in the previous layer 802.

[0084] The advantage of using convolutional layers 804 is that it can take advantage of the spatial local correlation of the input data by forcing a local connectivity pattern between nodes in neighboring layers, in particular by connecting each node only to a small region of nodes in the previous layer.

[0085] exist Figure 8In the illustrated embodiment, the input layer 802 includes 36 nodes 812 arranged as a two-dimensional 6×6 matrix. The convolutional layer 804 includes 72 nodes 814 arranged as two two-dimensional 6×6 matrices, each of which is the result of convolving the values ​​of the input layer with a kernel. Equivalently, the nodes 814 of the convolutional layer 804 can be interpreted as arranged as a three-dimensional 6×6×2 matrix, where the last dimension is the depth dimension.

[0086] The pooling layer 806 is characterized by the activation function of its node 816, which forms the pooling operation based on a nonlinear pooling function f, as well as the structure and weights of the incoming edges. For example, in the two-dimensional case, the value x of node 816 in pooling layer 806... (n) It can be based on the value x of node 814 of the previous layer 804. (n-1) And calculated as:

[0087] .

[0088] In other words, by using pooling layer 806, the number of nodes 814 and 816 can be reduced by replacing a number of neighboring nodes 814 (d1·d2) in the previous layer 804 with a single node 816, which is calculated based on the values ​​of the number of neighboring nodes. Specifically, the pooling function f can be a maximum function, an average function, or an L2 norm function. Also, for pooling layer 806, the weights of the incoming edges are fixed and are not modified through training.

[0089] The advantage of using pooling layer 806 is that it reduces the number of nodes 814 and 816, as well as the number of parameters. This leads to a reduction in the computational cost of the network and helps control overfitting.

[0090] exist Figure 8 In the illustrated embodiment, pooling layer 806 is max pooling, which replaces four adjacent nodes with only one node whose value is the maximum of the four adjacent node values. Max pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, max pooling is applied to each of the two 2D matrices, thereby reducing the number of nodes from 72 to 18.

[0091] The fully connected layer 808 is characterized by the fact that most, in particular all, edges between node 816 of the previous layer 806 and node 818 of the fully connected layer 808 exist, and the weight of each edge can be adjusted individually.

[0092] In this embodiment, the nodes 816 of the preceding layer 806 of the fully connected layer 808 are displayed both as a two-dimensional matrix and additionally as unrelated nodes (indicated as a row of nodes, where the number of nodes is reduced for better presentability). In this embodiment, the number of nodes 818 in the fully connected layer 808 is equal to the number of nodes 816 in the preceding layer 806. Alternatively, the number of nodes 816 and 818 can be different.

[0093] Furthermore, in this embodiment, the value of node 820 in output layer 810 is determined by applying the Softmax function to the value of node 818 in the previous layer 808. By applying the Softmax function, the sum of the values ​​of all nodes 820 in output layer 810 is 1, and all values ​​of all nodes 820 in output layer 810 are real numbers between 0 and 1.

[0094] The convolutional neural network 800 may also include ReLU (Rectified Linear Unit) layers or activation layers with nonlinear transfer functions. Specifically, the number and structure of nodes included in the ReLU layer are equivalent to the number and structure of nodes included in the previous layer. In particular, the value of each node in the ReLU layer is computed by applying the rectification function to the value of the corresponding node in the previous layer.

[0095] The inputs and outputs of different convolutional neural network blocks can be wired using summation (residual / dense neural networks), element-wise multiplication (attention), or other differentiable operators. Therefore, if the entire pipeline is differentiable, the convolutional neural network architecture can be nested rather than sequential.

[0096] Specifically, a convolutional neural network (CNN) 800 can be trained based on the backpropagation algorithm. To prevent overfitting, regularization methods can be used, such as dropout of nodes 812-820, random pooling, the use of artificial data, weight decay based on L1 or L2 norm, or maximum norm constraints. Different loss functions can be combined to train the same neural network to reflect the joint training objective. A subset of neural network parameters can be excluded from the optimization to retain weights pre-trained on a different dataset.

[0097] The systems, apparatus, and methods described herein can be implemented using digital circuitry or using one or more computers employing known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or be coupled to one or more mass storage devices, such as one or more disks, internal hard disks and removable disks, magneto-optical disks, optical disks, etc.

[0098] The systems, apparatus, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0099] The systems, apparatus, and methods described herein can be implemented within a network-based cloud computing system. In such a system, a server or another processor connected to the network communicates with one or more client computers via the network. Client computers can communicate with the server via, for example, a web browser application residing and operating on the client computer. Client computers can store data on the server and access that data via the network. Client computers can transmit requests for data or for online services to the server via the network. The server can perform the requested service and provide data to (one or more) client computers(s). The server can also transmit data adapted to cause client computers to perform specified functions (e.g., perform calculations, display specified data on a screen, etc.). For example, the server can transmit data adapted to cause client computers to perform one or more steps or functions of the methods and workflows described herein (including...). Figure 1 A request for one or more steps or functions). Certain steps or functions of the methods and workflows described herein (including...) Figure 1 One or more steps or functions of the methods and workflows described herein may be performed by a server or by another processor in a network-based cloud computing system. Figure 1 One or more steps) can be performed by a client computer in a web-based cloud computing system. The steps or functions of the methods and workflows described herein (including...) Figure 1 One or more steps can be performed by a server and / or by a client computer in a web-based cloud computing system in any combination.

[0100] The systems, apparatuses, and methods described herein can be implemented using a computer program product tangibly embodied in an information carrier (e.g., embodied in a non-transitory machine-readable storage device) for execution by a programmable processor; and the methods and workflow steps described herein (including Figure 1One or more steps or functions of a computer can be implemented using one or more computer programs that can be executed by such a processor. A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform an activity or produce a result. A computer program can be written in any form of programming language (including compiled or interpreted languages) and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0101] Figure 9 A high-level block diagram of an example computer 902, which can be used to implement the systems, apparatus, and methods described herein, is depicted. Computer 902 includes a processor 904 operatively coupled to a data storage device 912 and a memory 910. Processor 904 controls this operation by executing computer program instructions that define the overall operation of computer 902. The computer program instructions may be stored in the data storage device 912 or other computer-readable medium and loaded into memory 910 when execution of the computer program instructions is desired. Therefore, Figure 2 The methods and workflow steps or functions can be defined by computer program instructions stored in memory 910 and / or data storage device 912, and can be controlled by processor 904 that executes the computer program instructions. For example, the computer program instructions can be implemented to be programmed by those skilled in the art to perform... Figure 1 The methods, workflow steps, or functions are computer-executable code. Therefore, by executing computer program instructions, processor 904 performs... Figure 1 The computer 902 may include methods and workflow steps or functions. The computer 902 may also include one or more network interfaces 906 for communicating with other devices via a network. The computer 902 may also include one or more input / output devices 908 (e.g., monitor, keyboard, mouse, speaker, buttons, etc.) that enable a user to interact with the computer 902.

[0102] Processor 904 may include both general-purpose microprocessors and special-purpose microprocessors, and may be the sole processor of computer 902 or one of multiple processors. For example, processor 904 may include one or more central processing units (CPUs). Processor 904, data storage device 912 and / or memory 910 may include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs), supplemented by them, or incorporated therein.

[0103] Both data storage device 912 and memory 910 include tangible, non-transitory computer-readable storage media. Both data storage device 912 and memory 910 may include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more disk storage devices (e.g., internal hard disks and removable disks), magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices (e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital versatile disc read-only memory (DVD-ROM) discs), or other non-volatile solid-state memory devices.

[0104] Input / output device 908 may include peripheral devices such as printers, scanners, displays, etc. For example, input / output device 908 may include display devices (such as cathode ray tube (CRT) or liquid crystal display (LCD) monitors) for displaying information to a user, a keyboard, and pointing devices such as mice or trackballs through which the user can provide input to computer 902.

[0105] Image acquisition device 914 can be connected to computer 902 to input image data (e.g., medical images) into computer 902. It is possible to implement image acquisition device 914 and computer 902 as a single device. It is also possible for image acquisition device 914 and computer 902 to communicate wirelessly via a network. In a possible embodiment, computer 902 can be remotely located relative to image acquisition device 914.

[0106] Any or all of the systems and apparatus discussed herein may be implemented using one or more computers (such as Computer 902).

[0107] Those skilled in the art will recognize that actual computer or computer system implementations may have other structures and may include other components, and for illustrative purposes, Figure 9 It is a high-level representation of some components in this type of computer.

[0108] The foregoing specific embodiments should be understood in each respect as illustrative and exemplary, not restrictive, and the scope of the invention disclosed herein is not determined by these specific embodiments, but rather by the claims as interpreted under the full breadth permitted by patent law. It is to be understood that the embodiments shown and described herein merely illustrate the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention. Various other combinations of features can be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A computer-implemented method, comprising: Receive one or more input medical images of the patient's blood vessels; Multiple vascular assessment tasks for evaluating blood vessels are performed using a machine learning-based model trained on shared features extracted from the one or more input medical images, which leverages multi-task learning. The multiple vascular assessment tasks include segmentation of the reference wall of the blood vessel and segmentation of the lumen of the blood vessel from the one or more input medical images. The machine learning-based model is trained for segmentation of the reference wall of the blood vessel and segmentation of the lumen of the blood vessel based on regularization of the consistency between the segmentation results of the reference wall of the blood vessel and the segmentation results of the lumen of the blood vessel in areas without abnormalities or lesions. as well as Output the results of the multiple vascular assessment tasks.

2. The computer-implemented method of claim 1, wherein the machine learning-based model is trained for segmentation of the reference wall of the blood vessel based on regularization of anatomical tapering for the blood vessel.

3. The computer-implemented method of claim 1, wherein regularization of the consistency between the segmentation results of the reference wall of the vessel in the region without abnormalities or lesions and the segmentation results of the vessel lumen is based on ground truth labels for the abnormalities and ground truth labels for the lesions.

4. The computer-implemented method of claim 1, wherein the machine learning-based model is trained for the segmentation of reference walls of blood vessels based on regularization of the segmentation of reference walls in regions with abnormalities or lesions.

5. The computer-implemented method of claim 4, wherein the regularization of the segmentation of the reference wall in a region having anomalies or lesions is based on ground truth labels for the anomalies and ground truth labels for the lesions.

6. The computer-implemented method of claim 1, wherein the machine learning-based model is trained for stenosis grading based on regularization of the consistency between ground truth stenosis grading and stenosis grading based on vascular reference wall segmentation results and lumen segmentation results.

7. The computer-implemented method of claim 1, wherein the machine learning-based model is trained for the consistency between ground truth lumen segmentation and lumen segmentation results for segmentation of the lumen of blood vessels.

8. The computer-implemented method of claim 1, wherein the plurality of vascular assessment tasks further includes image-based stenosis grading of stenosis in the vascular vessels, and wherein the results of image-based stenosis grading, segmentation results of the reference wall, and segmentation results of the lumen are consistent.

9. The computer-implemented method according to claim 1, further comprising: The degree of stenosis in a blood vessel is determined by dividing the reference wall of the blood vessel and dividing the lumen of the blood vessel.

10. The computer-implemented method of claim 1, wherein performing multiple vascular assessment tasks for assessing blood vessels by utilizing a machine learning-based model trained based on shared features extracted from the one or more input medical images through multi-task learning comprises: Determine the uncertainty estimate for each of the plurality of vascular assessment tasks.

11. An apparatus comprising: Device for receiving one or more input medical images of a patient's blood vessels; An apparatus for performing multiple vascular assessment tasks for evaluating blood vessels using a machine learning-based model trained by leveraging multi-task learning based on shared features extracted from one or more input medical images, the multiple vascular assessment tasks including segmentation of a reference wall of a blood vessel from the one or more input medical images and segmentation of the lumen of a blood vessel from the one or more input medical images, wherein the machine learning-based model is trained for segmentation of the reference wall of a blood vessel and segmentation of the lumen of a blood vessel based on regularization of the consistency between the segmentation results of the reference wall of a blood vessel in regions without abnormalities or lesions and the segmentation results of the lumen of a blood vessel. as well as A device for outputting the results of the multiple vascular assessment tasks.

12. The device of claim 11, wherein the machine learning-based model is trained for segmentation of the reference wall of the blood vessel based on regularization of anatomical tapering for the blood vessel.

13. The device of claim 11, wherein the regularization of the consistency between the segmentation result of the reference wall of the vessel in the region without abnormality or lesion and the segmentation result of the lumen of the vessel is based on the ground truth label for the abnormality and the ground truth label for the lesion.

14. The device of claim 11, wherein the machine learning-based model is trained for segmentation of the reference wall of the blood vessel based on regularization of the segmentation of the reference wall in the region having an abnormality or lesion.

15. The device of claim 14, wherein the regularization of the segmentation of the reference wall in a region having anomalies or lesions is based on ground truth labels for the anomalies and ground truth labels for the lesions.

16. A non-transitory computer-readable medium storing computer program instructions, said computer program instructions, when executed by a processor, cause the processor to perform operations, said operations including: Receive one or more input medical images of the patient's blood vessels; Multiple vascular assessment tasks for evaluating blood vessels are performed using a machine learning-based model trained on shared features extracted from the one or more input medical images, which leverages multi-task learning. The multiple vascular assessment tasks include segmentation of the reference wall of the blood vessel and segmentation of the lumen of the blood vessel from the one or more input medical images. The machine learning-based model is trained for segmentation of the reference wall of the blood vessel and segmentation of the lumen of the blood vessel based on regularization of the consistency between the segmentation results of the reference wall of the blood vessel and the segmentation results of the lumen of the blood vessel in areas without abnormalities or lesions. as well as Output the results of the multiple vascular assessment tasks.

17. The non-transitory computer-readable medium of claim 16, wherein the machine learning-based model is trained for stenosis segmentation of the reference wall and the lumen of the blood vessel based on regularization of the consistency between ground truth stenosis grading and stenosis grading based on segmentation results of the reference wall and the lumen of the blood vessel.

18. The non-transitory computer-readable medium of claim 16, wherein the machine learning-based model is trained for the consistency between ground truth lumen segmentation and lumen segmentation results for the segmentation of vascular lumens.

19. The non-transitory computer-readable medium of claim 16, wherein the plurality of vascular assessment tasks further includes image-based stenosis grading of stenosis in the vascular vessels, and wherein the results of image-based stenosis grading, segmentation results of the reference wall, and segmentation results of the lumen are consistent.

20. The non-transitory computer-readable medium of claim 16, wherein the operation further comprises: The degree of stenosis in a blood vessel is determined by dividing the reference wall of the blood vessel and dividing the lumen of the blood vessel.

21. The non-transitory computer-readable medium of claim 16, wherein performing multiple vascular assessment tasks for assessing blood vessels by utilizing a machine learning-based model trained on shared features extracted from the one or more input medical images through multi-task learning comprises: Determine the uncertainty estimate for each of the plurality of vascular assessment tasks.

Images