AI-based medical imaging analysis of photon count data

Abstract

Systems and methods for performing medical imaging analysis tasks from PCCT (Photon Counting Computed Tomography) imaging data are provided. The PCCT imaging data acquired from the PCCT imaging device is received. A plurality of PCCT virtual images are generated from the PCCT imaging data. A plurality of medical imaging analysis subtasks are performed based on the plurality of PCCT virtual images using a plurality of machine learning based networks. The results of the medical imaging analysis subtasks are combined to generate results of the medical imaging analysis task. And outputting the result of the medical imaging analysis task.

Description

Technical Field

[0001] The present invention relates generally to medical imaging analysis, and in particular to AI (artificial intelligence) / ML (machine learning) based medical imaging analysis of photon counting data. Background Art

[0002] Photon counting is a technique in CT (Computed Tomography) imaging in which spectral imaging data is acquired by counting individual photons using an energy-selective photon counting detector. Images with high spatial resolution, no electronic noise, improved contrast-to-noise ratio, and spectral information can be generated from the spectral imaging data.

[0003] Recently, artificial intelligence and machine learning based systems have been proposed for performing various medical imaging analysis tasks on medical images. However, conventional artificial intelligence and machine learning based systems are not suitable for processing images generated from spectral imaging data acquired via photon counting. Summary of the invention

[0004] According to one or more embodiments, a system and method for performing a medical imaging analysis task from PCCT (photon counting computed tomography) imaging data is provided. PCCT imaging data acquired from a PCCT imaging device is received. A plurality of PCCT virtual images are generated from the PCCT imaging data. A plurality of medical imaging analysis subtasks are performed based on the plurality of PCCT virtual images using a plurality of machine learning-based networks. The results of the medical imaging analysis subtasks are combined to generate a result of the medical imaging analysis task. The result of the medical imaging analysis task is output.

[0005] In one embodiment, the plurality of PCCT virtual images include at least one of a virtual monoenergetic image, a virtual non-contrast image, a virtual iodine image, a virtual pure lumen image, and an ultra-high resolution image.

[0006] In one embodiment, the results of the medical imaging analysis subtasks are combined based on statistical weightings of the results of the medical imaging analysis subtasks. In another embodiment, the results of the medical imaging analysis subtasks are combined based on learned weightings of the results of the medical imaging analysis subtasks. The learned weightings are learned based on ground truth data. In another embodiment, the results of the medical imaging analysis subtasks are combined based on context-based weightings of the results of the medical imaging analysis subtasks.

[0007] In one embodiment, the medical imaging analysis task includes automatic reporting of coronary artery stenosis, and the plurality of medical imaging analysis subtasks include at least one of detection of coronary artery centerline, detection of stenosis, grading of stenosis, marking of vessel segments, and detection of lumen and plaque.

[0008] In one embodiment, the medical imaging analysis tasks include at least one of automated CAD-RADS (Coronary Artery Disease Reporting and Data System) scoring, detection and quantification of coronary plaque and fat, calculation of CT-FFR (Computed Tomography Fractional Flow Reserve), detection of stents and quantification of in-stent restenosis, and detection of bypass grafts and assessment of graft patency, and multiple medical imaging analysis subtasks include at least one of coronary centerline tracking, lesion detection, segment labeling, lumen and outer wall segmentation, and quantification of plaque components.

[0009] These and other advantages of the present invention will become apparent to those of ordinary skill in the art by reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A method for performing medical imaging analysis tasks from PCCT imaging data according to one or more embodiments is shown; Figure 2 A workflow for performing medical imaging analysis tasks from PCCT imaging data according to one or more embodiments is shown; Figure 3 shows exemplary lumen segmentation results generated from different PCCT virtual images according to one or more embodiments; Figure 4 An exemplary artificial neural network that can be used to implement one or more embodiments is shown; Figure 5 shows a convolutional neural network that can be used to implement one or more embodiments; and Figure 6 A high-level block diagram of a computer that can be used to implement one or more embodiments is shown. DETAILED DESCRIPTION

[0011] The present invention generally relates to methods and systems for AI / ML-based medical imaging analysis of photon counting data. Embodiments of the present invention are described herein to give a visual understanding of such methods and systems. A digital image is typically composed of digital representations of one or more objects (or shapes). Digital representations of objects are generally described herein in terms of identifying and manipulating objects. Such manipulations are virtual manipulations performed in the memory or other circuits / hardware of a computer system. Therefore, it is to be understood that embodiments of the present invention may be performed within a computer system using data stored within the computer system. In addition, references to image pixels herein may equally refer to voxels of an image, and vice versa.

[0012] The embodiments described herein provide an AI system for performing medical imaging analysis tasks on multiple PCCT virtual images. The AI ​​system includes multiple machine learning-based networks for performing multiple medical imaging analysis subtasks based on multiple PCCT virtual images. The multiple PCCT virtual images can be of different types, such as, for example, monoenergetic images reconstructed from different energy levels, non-contrast images, iodine images, pure lumen images, ultra-high resolution images, etc. The results of the medical imaging analysis subtasks are combined to generate the results of the medical imaging analysis task. Advantageously, the embodiments described herein process multiple PCCT virtual images, thereby improving the performance of the medical imaging analysis task.

[0013] Figure 1 A method 100 for performing medical imaging analysis tasks from PCCT imaging data according to one or more embodiments is shown. The steps of the method 100 may be performed by one or more suitable computing devices (such as, for example, Figure 6 computer 602) to execute. Figure 2 A workflow 200 for performing medical imaging analysis tasks from PCCT imaging data is shown in accordance with one or more embodiments. Figure 1 and Figure 2 Will be described together.

[0014] exist Figure 1 At step 102, PCCT imaging data acquired from a PCCT imaging device is received. The PCCT imaging device is equipped with a photon counting detector, which is used to count the number of incident photons from x-rays and directly measure the photon energy. The PCCT imaging data can be imaging data of any anatomical object or object of interest (such as, for example, an organ, bone, lesion, etc.) of the patient. In one example, Figure 2 As shown in the workflow 200 of FIG. 1 , the PCCT imaging data may be PCCT imaging data 202. The PCCT imaging data may be received directly from a PCCT imaging device as the PCCT imaging data is acquired, may be received by loading previously acquired PCCT imaging data from a storage device or memory of a computer system, or may be received by receiving the PCCT imaging data from a remote computer system.

[0015] exist Figure 1 At step 104, a plurality of PCCT virtual images are generated from the PCCT imaging data. In one example, Figure 2As shown in the workflow 200 of FIG. 2 , the plurality of PCCT virtual images is a virtual image 204. The plurality of PCCT virtual images may be generated from the PCCT imaging data using any suitable (e.g., known) method. For example, a virtual non-contrast, iodine, or pure luminal image may be reconstructed by decomposing materials such as iodine, calcium, and fat, or a virtual monoenergetic image may be reconstructed based on weighting and combining different energy bins available in the spectral data. In addition, the small pixel size of the photon counting detector allows for the acquisition of ultra-high resolution images having a slice thickness that is significantly less than conventional CT.

[0016] The multiple PCCT virtual images can be of different types. For example, the multiple PCCT virtual images can include virtual monoenergetic images reconstructed from different energy levels (e.g., 50keV or 100keV (kiloelectron volts)), virtual non-contrast images, virtual iodine images, virtual pure luminal images with calcium subtracted from contrast-enhanced scans, ultra-high resolution images, or any other suitable type of PCCT virtual images.

[0017] In one embodiment, multiple PCCT virtual images may be enriched based on various acquisition and reconstruction protocol parameters, such as, for example, reconstruction algorithms and convolution kernels.

[0018] exist Figure 1 At step 106, multiple machine learning-based networks are used to perform multiple medical imaging analysis subtasks based on multiple PCCT virtual images. Figure 2 As shown in the workflow 200, the plurality of machine learning-based networks is an ML-based network 206 that receives a virtual image 204 as an input and generates an AI result 208 as an output. At least some of the plurality of medical imaging analysis subtasks may be performed by different machine learning-based networks and / or the same machine learning-based network. The plurality of medical imaging analysis subtasks may include any suitable imaging analysis tasks, such as, for example, segmentation, detection, registration, and the like.

[0019] Figure 3 3 shows exemplary lumen segmentation results 300 generated from different PCCT virtual images according to one or more embodiments. The lumen segmentation results 300 are generated by a machine learning based segmentation network that performs a medical imaging analysis subtask of lumen segmentation. The top row shows PCCT virtual images, and the bottom row shows the corresponding lumen segmentation results. Figure 3 As can be seen in the figure, the PCCT virtual image includes VCE (virtual iodine image) and VME (virtual monoenergetic image) reconstructed from 40keV to 190keV energy levels.

[0020] Reference back Figure 1, multiple machine learning-based networks can be part of an AI system. The AI ​​system is configured to perform one or more overarching medical imaging analysis tasks by applying each of the multiple machine learning-based networks to perform a corresponding one of the medical imaging analysis subtasks. In one embodiment, the medical imaging analysis task includes automatic reporting of coronary artery stenosis, and the multiple medical imaging analysis subtasks include detection of coronary artery centerline, detection of stenosis, grading of stenosis, marking of vascular segments, detection of lumen and plaque, etc. In another embodiment, in the context of coronary artery applications, the medical imaging analysis task includes at least one of automated CAD-RADS (Coronary Artery Disease Reporting and Data System) scoring, detection and quantification of coronary artery plaque and fat, calculation of CT-FFR (Computed Tomography Blood Flow Reserve Fraction), detection of stents and quantification of in-stent restenosis, and detection of bypass grafts and assessment of graft patency, and the multiple medical imaging analysis subtasks include coronary artery centerline tracking, lesion detection, segment marking, lumen and outer wall segmentation, and quantification of plaque components.

[0021] Each machine learning-based network receives one or more of the plurality of PCCT virtual images as input and generates a result of the medical imaging analysis subtask as output. In some embodiments, the machine learning-based network may additionally receive one or more outputs generated by one or more other machine learning-based networks in the plurality of machine learning-based networks as input. The plurality of machine learning-based networks are trained using any suitable method during a previous offline or training phase. Once trained, the plurality of machine learning-based networks are used in an online or inference phase (e.g., during Figure 1 ) is applied to perform multiple medical imaging analysis subtasks.

[0022] In one embodiment, instead of multiple machine learning based networks, the AI ​​system may include non-machine learning based components for performing one or more of the multiple medical imaging analysis subtasks.

[0023] exist Figure 1 At step 108, the results of the medical imaging analysis subtasks are combined to generate the result of the medical imaging analysis task. Figure 2 As shown in the workflow 200, the AI ​​results 208 of the medical imaging analysis subtasks are combined 210 to generate a final result 212 of the medical imaging analysis task. The results of the medical imaging analysis subtasks may be combined according to any suitable method.

[0024] In one embodiment, the results of the medical imaging analysis subtasks are combined into a single output that produces the best performance on the selected medical imaging analysis task. For each individual machine learning-based network, the combination can be performed at the AI ​​system level or at the component level.

[0025] In one embodiment, the results of the medical imaging analysis subtasks are combined based on a weighting scheme. In one embodiment, the results of the medical imaging analysis subtasks are combined based on statistical weightings (such as, for example, mean, median, mode, etc.). In another embodiment, the results of the medical imaging analysis subtasks are combined based on learned weightings. In this embodiment, the weights of the optimized target objects are determined based on ground truth data annotated by experts. For example, for the coronary artery use case, the ground truth data may include other modalities, such as, for example, coronary angiogram, OCT (optical coherence tomography), IVUS (intravascular ultrasound), etc. In a further embodiment, the results of the medical imaging analysis subtasks are combined via context-based weighting. In this embodiment, a specific weighting scheme can be selected based on disease characteristics or other contextual information. For example, different weighting schemes can be used for segmentation based on plaque components (calcified or soft). In a further embodiment, the results of the medical imaging analysis subtasks are combined via user-defined weightings.

[0026] exist Figure 1 At step 110, the result of the medical imaging analysis task is output. For example, the result of the medical imaging analysis task may be output by displaying the result of the medical imaging analysis task on a display device of the computer system, storing the result of the medical imaging analysis task on a memory or storage device of the computer system, or transmitting the result of the medical imaging analysis task to a remote computer system.

[0027] In one embodiment, Figure 1 At step 110, based on a medical imaging analysis task (e.g., stenosis grading vs. plaque quantification) or a disease characteristic (e.g., calcified plaque vs. soft plaque), one or more of the plurality of PCCT virtual images may be displayed or presented to the user. For example, one or more of the most weighted PCCT virtual images may be selected for display to the user.

[0028] In one embodiment, the Figure 1 A composite image including the results of the medical imaging analysis subtask is displayed to the user at step 110. The composite image can be used to convey the results of the medical imaging analysis task of the AI ​​system or for diagnostic evaluation.

[0029] In one embodiment, Figure 1 At step 106, about Figure 1 The weighting scheme described in step 108 can be directly applied to combine or fuse multiple PCCT virtual images. The fused images can be input to multiple machine learning based networks to perform multiple medical imaging analysis subtasks.

[0030] In one embodiment, a plurality of machine learning based networks may be run at an operating point that may be estimated at run time by analyzing the output of the machine learning based networks for one or more of a plurality of PCCT virtual images.

[0031] The embodiments described herein are described with respect to the claimed system and the claimed method. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. In other words, the features described or claimed in the context of the method may be utilized to improve the claims for the system. In this case, the functional features of the method are embodied by providing the target unit of the system.

[0032] In addition, certain embodiments described herein are described with respect to methods and systems for utilizing trained machine learning-based models, and with respect to methods and systems for training machine learning-based models. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. In other words, features described or claimed in the context of methods and systems for utilizing trained machine learning-based models may be utilized to improve claims for methods and systems for training machine learning-based models, and vice versa.

[0033] In particular, the trained machine learning-based models applied in the embodiments described herein can be adapted by methods and systems for training machine learning-based models. In addition, the input data of the trained machine learning-based models can include advantageous features and embodiments of the training input data, and vice versa. In addition, the output data of the trained machine learning-based models can include advantageous features and embodiments of the output training data, and vice versa.

[0034] Generally speaking, trained machine learning-based models mimic the cognitive functions of humans in connection with other human minds. In particular, through training based on training data, trained machine learning-based models are able to adapt to new situations and detect and extrapolate patterns.

[0035] In general, the parameters of the machine learning based model can be adapted by means of training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning and / or active learning can be used. In addition, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of the trained machine learning based model can be iteratively adapted through several training steps.

[0036] In particular, the trained machine learning based model may include a neural network, a support vector machine, a decision tree and / or a Bayesian network, and / or the trained machine learning based model may be based on k-means clustering, Q learning, a genetic algorithm and / or an association rule. In particular, the neural network may be a deep neural network, a convolutional neural network or a convolutional deep neural network. In addition, the neural network may be an adversarial network, a deep adversarial network and / or a generative adversarial network.

[0037] Figure 4 An embodiment of an artificial neural network 400 according to one or more embodiments is shown. Alternative terms for "artificial neural network" are "neural network", "artificial neural network" or "neural network". Artificial neural network 400 can be used to implement the machine learning networks described herein, such as, for example, in Figure 1 The plurality of machine learning based networks or Figure 2 ML-based network 206.

[0038] The artificial neural network 400 includes nodes 402-422 and edges 432, 434, ... 436, wherein each edge 432, 434, ... 436 is a directed connection from a first node 402-422 to a second node 402-422. Generally speaking, the first node 402-422 and the second node 402-422 are different nodes 402-422, and it is also possible that the first node 402-422 and the second node 402-422 are the same. For example, in Figure 4 , edge 432 is a directed connection from node 402 to node 406, and edge 434 is a directed connection from node 404 to node 406. Edges 432, 434, ... 436 from a first node 402-422 to a second node 402-422 are also denoted as "incoming edges" to the second node 402-422, and as "outgoing edges" to the first node 402-422.

[0039] In this embodiment, the nodes 402-422 of the artificial neural network 400 may be arranged in layers 424-430, wherein the layers may include an inherent order introduced by edges 432, 434, ... 436 between the nodes 402-422. In particular, the edges 432, 434, ... 436 may only exist between adjacent layers of nodes. Figure 4In the embodiment shown in FIG. 4 , there is an input layer 424 that includes only nodes 402 and 404 without incoming edges, an output layer 430 that includes only nodes 422 without outgoing edges, and hidden layers 426, 428 between the input layer 424 and the output layer 430. In general, the number of hidden layers 426, 428 can be selected arbitrarily. The number of nodes 402 and 404 within the input layer 424 is generally related to the number of input values ​​of the neural network 400, and the number of nodes 422 within the output layer 430 is generally related to the number of output values ​​of the neural network 400.

[0040] In particular, a (real) number may be assigned as a value to each node 402-422 of the neural network 400. Here, x (n) i represents the value of the i-th node 402-422 of the n-th layer 424-430. The values ​​of the nodes 402-422 of the input layer 424 are equivalent to the input values ​​of the neural network 400, and the value of the node 422 of the output layer 430 is equivalent to the output value of the neural network 400. In addition, each edge 432, 434, ... 436 may include a weight as a real number, in particular, the weight is a real number in the interval [-1, 1] or in the interval [0, 1]. Here, w (m,n) i,j represents the weight of the edge between the i-th node 402-422 of the m-th layer 424-430 and the j-th node 402-422 of the n-th layer 424-430. (n) i,j is defined for the weight w (n,n+1) i,j .

[0041] In particular, in order to calculate the output value of the neural network 400, the input value is propagated through the neural network. In particular, the values ​​of the nodes 402-422 of the (n+1)th layer 424-430 can be calculated based on the values ​​of the nodes 402-422 of the nth layer 424-430 by the following formula:

[0042] In this article, function f is a transfer function (another term is "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logistic function, generalized logistic function, hyperbolic tangent function, inverse tangent function, error function, smooth step function), or rectifier functions. Transfer functions are mainly used for normalization purposes.

[0043] In particular, these values ​​are propagated through the neural network layer by layer, where the value of the input layer 424 is given by the input of the neural network 400, where the value of the first hidden layer 426 can be calculated based on the value of the input layer 424 of the neural network, where the value of the second hidden layer 428 can be calculated based on the value of the first hidden layer 426, and so on.

[0044] To set the value of edge w (m,n) i,j , training data must be used to train the neural network 400. In particular, the training data includes training input data and training output data (denoted as t i ). For the training step, the neural network 400 is applied to the training input data to generate the calculated output data. In particular, the training data and the calculated output data include a certain number of values, which is equal to the number of nodes of the output layer.

[0045] In particular, the comparison between the calculated output data and the training data is used to recursively adapt the weights within the neural network 400 (back propagation algorithm). In particular, the weights are changed according to the following formula: where γ is the learning rate and if the (n+1)th layer is not the output layer, it can be based on δ (n+1) j Recursively add the number δ (n) j Calculated as: And if the (n+1)th layer is the output layer 430, it is calculated as: where f' is the first derivative of the activation function, and y (n+1) j is the comparison training value of the j-th node of the output layer 430.

[0046] Figure 5 A convolutional neural network 500 is shown in accordance with one or more embodiments. The convolutional neural network 500 may be used to implement the machine learning networks described herein, such as, for example, in Figure 1 The multiple machine learning based networks used in step 106 or Figure 2 ML-based network 206.

[0047] exist Figure 5In the embodiment shown in , the convolutional neural network 500 includes an input layer 502, a convolutional layer 504, a pooling layer 506, a fully connected layer 508, and an output layer 510. Alternatively, the convolutional neural network 500 may include several convolutional layers 504, several pooling layers 506, and several fully connected layers 508, as well as other types of layers. The order of these layers can be arbitrarily selected, and the fully connected layer 508 is usually used as the last layer before the output layer 510.

[0048] In particular, within the convolutional neural network 500, the nodes 512-520 of a layer 502-510 may be considered to be arranged as a d-dimensional matrix or a d-dimensional image. In particular, in the two-dimensional case, the values ​​of the nodes 512-520 indexed by i and j in the nth layer 502-510 may be represented as x (n) [i,j] However, the arrangement of the nodes 512-520 of a layer 502-510 has no effect on the computations performed within the convolutional neural network 500 itself, since they are given only by the structure and weights of the edges.

[0049] In particular, the convolution layer 504 is characterized by the structure and weights of the incoming edges that form the convolution operation based on a certain number of kernels. In particular, the structure and weights of the incoming edges are selected so that the value x of the node 514 of the convolution layer 504 is (n) k Based on the value x of node 512 of the previous layer 502 (n-1) is calculated as the convolution x (n) k =K k *x (n-1) , where the convolution* in two dimensions is defined as:

[0050] Here, the kth kernel K k Is a d-dimensional matrix (a two-dimensional matrix in this embodiment), which is typically small compared to the number of nodes 512-518 (e.g., a 3×3 matrix or a 5×5 matrix). In particular, this means that the weights of the incoming edges are not independent, but are selected so that they produce the convolution formula. In particular, for the kernel as a 3×3 matrix, there are only 9 independent weights (each entry in the kernel matrix corresponds to an independent weight), regardless of the number of nodes 512-520 in the corresponding layer 502-510. In particular, for a convolutional layer 504, the number of nodes 514 in the convolutional layer is equal to the number of nodes 512 in the previous layer 502 multiplied by the number of kernels.

[0051] If the nodes 512 of the previous layer 502 are arranged as a d-dimensional matrix, the use of multiple kernels can be interpreted as adding an additional dimension (denoted as the "depth" dimension) so that the nodes 514 of the convolution layer 514 are arranged as a (d+1)-dimensional matrix. If the nodes 512 of the previous layer 502 are already arranged as a (d+1)-dimensional matrix including the depth dimension, the use of multiple kernels can be interpreted as expanding along the depth dimension so that the nodes 514 of the convolution layer 504 are also arranged as a (d+1)-dimensional matrix, where the size of the (d+1)-dimensional matrix with respect to the depth dimension is the number of kernels times the size in the previous layer 502.

[0052] An advantage of using convolutional layers 504 is that spatially local correlations in the input data can be exploited by enforcing local connectivity patterns between nodes in adjacent layers, in particular by connecting each node only to a small region of nodes in the previous layer.

[0053] exist Figure 5 In the embodiment shown in , the input layer 502 includes 36 nodes 512 arranged as a two-dimensional 6×6 matrix. The convolution layer 504 includes 72 nodes 514 arranged as two two-dimensional 6×6 matrices, each of which is the result of convolving the values ​​of the input layer with the kernel. Equivalently, the nodes 514 of the convolution layer 504 can be interpreted as being arranged as a three-dimensional 6×6×2 matrix, where the last dimension is the depth dimension.

[0054] The pooling layer 506 is characterized by the activation function of its nodes 516 and the structure and weights of the incoming edges that form the pooling operation based on the nonlinear pooling function f. For example, in the two-dimensional case, the value x of the node 516 of the pooling layer 506 ( n ) It can be based on the value x of the node 514 of the previous layer 504 ( n- 1) and is calculated as: x (n) [i,j]=f(x (n-1) [id 1 ,jd 2 ],...,x (n-1) [id 1 +d 1 -1,jd 2 +d 2 -1]).

[0055] In other words, by using the pooling layer 506, the number of nodes 514, 516 can be reduced by replacing the number d1·d2 of neighboring nodes 514 in the previous layer 504 with a single node 516 in the pooling layer, which is calculated based on the values ​​of the number of neighboring nodes. In particular, the pooling function f can be a maximum function, an average value, or an L2-norm. In particular, for the pooling layer 506, the weights of the incoming edges are fixed and are not modified by training.

[0056] The advantage of using the pooling layer 506 is that the number of nodes 514, 516 and the number of parameters are reduced. This results in a reduction in the amount of computation in the network and in control of overfitting.

[0057] exist Figure 5 In the embodiment shown in , pooling layer 506 is maximum pooling, replacing four neighboring nodes with only one node whose value is the maximum of the values ​​of the four neighboring nodes. Maximum pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, maximum pooling is applied to each of the two two-dimensional matrices, thereby reducing the number of nodes from 72 to 18.

[0058] The fully connected layer 508 is characterized by the fact that most of the edges, in particular all of the edges, between the nodes 516 of the previous layer 506 and the nodes 518 of the fully connected layer 508 exist, and the weight of each of the edges can be adjusted individually.

[0059] In this embodiment, the nodes 516 of the previous layer 506 of the fully connected layer 508 are displayed both as a two-dimensional matrix and additionally as unrelated nodes (which are indicated as a row of nodes with a reduced number of nodes for better presentability). In this embodiment, the number of nodes 518 in the fully connected layer 508 is equal to the number of nodes 516 in the previous layer 506. Alternatively, the number of nodes 516, 518 may be different.

[0060] Furthermore, in this embodiment, the values ​​of the nodes 520 of the output layer 510 are determined by applying the Softmax function to the values ​​of the nodes 518 of the previous layer 508. By applying the Softmax function, the sum of the values ​​of all the nodes 520 of the output layer 510 is 1, and all the values ​​of all the nodes 520 of the output layer are real numbers between 0 and 1.

[0061] The convolutional neural network 500 may also include a ReLU (rectified linear unit) layer or an activation layer having a nonlinear transfer function. In particular, the number of nodes and the structure of the nodes included in the ReLU layer are equivalent to the number of nodes and the structure of the nodes included in the previous layer. In particular, the value of each node in the ReLU layer is calculated by applying a rectification function to the value of the corresponding node of the previous layer.

[0062] The inputs and outputs of different CNN blocks can be wired using summation (residual / dense NNs), element-wise multiplication (attention), or other differentiable operators. Therefore, if the entire pipeline is differentiable, the CNN architecture can be nested instead of sequential.

[0063] In particular, the convolutional neural network 500 can be trained based on a back-propagation algorithm. To prevent overfitting, regularization methods such as dropout of nodes 512-520, random pooling, use of artificial data, weight decay based on L1 or L2 norms, or maximum norm constraints can be used. Different loss functions can be combined for training the same neural network to reflect joint training objectives. A subset of neural network parameters can be excluded from optimization to retain weights pre-trained on another data set.

[0064] The systems, devices, and methods described herein may be implemented using digital circuits, or may be implemented using one or more computers using known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or may be coupled to one or more mass storage devices, such as one or more disks, internal hard disks and removable disks, magneto-optical disks, optical disks, and the like.

[0065] The systems, devices, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0066] The systems, devices, and methods described herein can be implemented within a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor connected to the network communicates with one or more client computers via the network. The client computer can communicate with the server via, for example, a web browser application that resides and operates on the client computer. The client computer can store data on the server and access the data via the network. The client computer can transmit a request for data or a request for an online service to the server via the network. The server can perform the requested service and provide the data to (one or more) client computers. The server can also transmit data adapted to cause the client computer to perform a specified function (e.g., perform a calculation, display specified data on a screen, etc.). For example, the server can transmit data adapted to cause the client computer to perform one or more steps or functions (including Figure 1 or 2) of one or more steps or functions). Certain steps or functions of the methods and workflows described herein (including Figure 1 One or more steps or functions of 2) can be performed by a server or by another processor in a network-based cloud computing system. Certain steps or functions of the methods and workflows described herein (including Figure 1 One or more steps of 2 or 3) can be performed by a client computer in a network-based cloud computing system. The steps or functions of the methods and workflows described herein (including Figure 1 Or one or more steps of 2) can be performed by a server and / or by a client computer in a network-based cloud computing system in any combination.

[0067] The systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier (e.g., embodied in a non-transitory machine-readable storage device) for execution by a programmable processor; and the methods and workflow steps described herein (including Figure 1 One or more steps or functions of the above-mentioned processing unit (or 2) can be implemented using one or more computer programs executable by such a processor. A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language (including compiled or interpreted languages), and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0068] Figure 6A high-level block diagram of an example computer 602 that can be used to implement the systems, apparatus, and methods described herein is depicted in FIG. The computer 602 includes a processor 604 operably coupled to a data storage device 612 and a memory 610. The processor 604 controls the overall operation of the computer 602 by executing computer program instructions that define such operation. The computer program instructions may be stored in the data storage device 612, or other computer-readable medium, and loaded into the memory 610 when execution of the computer program instructions is desired. Thus, Figure 1 The method and workflow steps or functions of the method 602 or 603 may be defined by computer program instructions stored in the memory 610 and / or the data storage device 612 and may be controlled by the processor 604 executing the computer program instructions. For example, the computer program instructions may be implemented as a program programmed by a person skilled in the art to execute Figure 1 or 2 method and workflow steps or functions of the computer executable code. Therefore, by executing the computer program instructions, the processor 604 performs Figure 1 The computer 602 may also include one or more network interfaces 606 for communicating with other devices via a network. The computer 602 may also include one or more input / output devices 608 (e.g., a display, keyboard, mouse, speaker, buttons, etc.) that enable a user to interact with the computer 602.

[0069] The processor 604 may include both general-purpose and special-purpose microprocessors, and may be the sole processor or one of multiple processors of the computer 602. For example, the processor 604 may include one or more central processing units (CPUs). The processor 604, the data storage device 612, and / or the memory 610 may include, be supplemented by, or be incorporated in one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs).

[0070] The data storage device 612 and the memory 610 each include a tangible, non-transitory computer-readable storage medium. The data storage device 612 and the memory 610 may each include a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices (such as an internal hard disk and a removable magnetic disk), a magneto-optical disk storage device, an optical disk storage device, a flash memory device, a semiconductor memory device (such as an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM), a digital versatile disk read-only memory (DVD-ROM) disk) or other non-volatile solid-state storage devices.

[0071] Input / output devices 608 may include peripheral devices such as printers, scanners, display screens, etc. For example, input / output devices 608 may include a display device (such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to a user, a keyboard, and a pointing device such as a mouse or trackball through which a user can provide input to the computer 602.

[0072] The image acquisition device 614 can be connected to the computer 602 to input image data (e.g., medical images) to the computer 602. It is possible to implement the image acquisition device 614 and the computer 602 as one device. It is also possible that the image acquisition device 614 and the computer 602 communicate wirelessly via a network. In a possible embodiment, the computer 602 can be remotely located relative to the image acquisition device 614.

[0073] One or more computers, such as computer 602, may be used to implement any or all of the systems and apparatus discussed herein.

[0074] Those skilled in the art will recognize that actual computer or computer system implementations may have other structures and may include other components, and for illustrative purposes, Figure 6 is a high-level representation of some of the components in such a computer.

[0075] Independent of the use of the grammatical term, individuals with masculine or feminine identities are also included in the term.

[0076] The foregoing specific embodiments should be understood in every respect to be illustrative and exemplary, rather than restrictive, and the scope of the invention disclosed herein is not determined by the specific embodiments, but by the claims interpreted according to the full breadth allowed by the patent law. It is to be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Various other feature combinations may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A computer-implemented method comprising: receiving PCCT imaging data acquired from a PCCT (Photon Counting Computed Tomography) imaging device; generating a plurality of PCCT virtual images from the PCCT imaging data; performing a plurality of medical imaging analysis subtasks based on the plurality of PCCT virtual images using a plurality of machine learning-based networks; combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task; as well as The result of the medical imaging analysis task is output.

2. The computer-implemented method of claim 1, wherein the plurality of PCCT virtual images comprises at least one of a virtual monoenergetic image, a virtual non-contrast image, a virtual iodine image, a virtual pure lumen image, and an ultra-high resolution image.

3. The computer-implemented method of claim 1 , wherein combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task comprises: The results of the medical imaging analysis subtasks are combined based on statistical weighting of the results of the medical imaging analysis subtasks.

4. The computer-implemented method of claim 1 , wherein combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task comprises: The results of the medical imaging analysis subtasks are combined based on learned weights of the results of the medical imaging analysis subtasks, the learned weights being learned based on ground truth data.

5. The computer-implemented method of claim 1 , wherein combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task comprises: The results of the medical imaging analysis subtasks are combined based on context-based weighting of the results of the medical imaging analysis subtasks.

6. The computer-implemented method of claim 1 , wherein the medical imaging analysis task comprises automatic reporting of coronary artery stenosis, and the plurality of medical imaging analysis subtasks comprises at least one of detection of coronary artery centerline, detection of stenosis, grading of stenosis, marking of vessel segments, and detection of lumen and plaque.

7. A computer-implemented method according to claim 1, wherein the medical imaging analysis tasks include at least one of automated CAD-RADS (Coronary Artery Disease Reporting and Data System) scoring, detection and quantification of coronary plaque and fat, calculation of CT-FFR (Computed Tomography Fractional Flow Reserve), detection of stents and quantification of in-stent restenosis, and detection of bypass grafts and assessment of graft patency, and the multiple medical imaging analysis subtasks include at least one of coronary artery centerline tracking, lesion detection, segment labeling, lumen and outer wall segmentation, and quantification of plaque composition.

8. A device comprising: means for receiving PCCT imaging data acquired from a PCCT (Photon Counting Computed Tomography) imaging device; means for generating a plurality of PCCT virtual images from said PCCT imaging data; means for performing a plurality of medical imaging analysis subtasks based on the plurality of PCCT virtual images using a plurality of machine learning based networks; means for combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task; as well as Means for outputting the results of said medical imaging analysis task. 9 . The apparatus according to claim 8 , wherein the plurality of PCCT virtual images include at least one of a virtual monoenergetic image, a virtual non-contrast image, a virtual iodine image, a virtual pure lumen image, and an ultra-high resolution image.

10. The apparatus according to claim 8, wherein the means for combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task comprises: Means for combining the results of the medical imaging analysis subtasks based on statistical weighting of the results of the medical imaging analysis subtasks.

11. The apparatus according to claim 8, wherein the means for combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task comprises: Means for combining the results of the medical imaging analysis subtasks based on learned weights of the results of the medical imaging analysis subtasks, the learned weights being learned based on ground truth data.

12. The apparatus according to claim 8, wherein the means for combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task comprises: Means for combining results of the medical imaging analysis subtasks based on context-based weighting of the results of the medical imaging analysis subtasks.

13. The apparatus of claim 8, wherein the medical imaging analysis task comprises automatic reporting of coronary artery stenosis, and the plurality of medical imaging analysis subtasks comprises at least one of detection of coronary artery centerline, detection of stenosis, grading of stenosis, marking of vessel segments, and detection of lumen and plaque.

14. The apparatus of claim 8, wherein the medical imaging analysis tasks include at least one of automated CAD-RADS (Coronary Artery Disease Reporting and Data System) scoring, detection and quantification of coronary plaque and fat, calculation of CT-FFR (Computed Tomography Fractional Flow Reserve), detection of stents and quantification of in-stent restenosis, and detection of bypass grafts and assessment of graft patency, and the plurality of medical imaging analysis subtasks include at least one of coronary artery centerline tracking, lesion detection, segment labeling, lumen and outer wall segmentation, and quantification of plaque composition.

15. A non-transitory computer readable medium storing computer program instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving PCCT imaging data acquired from a PCCT (Photon Counting Computed Tomography) imaging device; generating a plurality of PCCT virtual images from the PCCT imaging data; performing a plurality of medical imaging analysis subtasks based on the plurality of PCCT virtual images using a plurality of machine learning-based networks; combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task; as well as The result of the medical imaging analysis task is output. 16 . The non-transitory computer readable medium of claim 15 , wherein the plurality of PCCT virtual images include at least one of a virtual monoenergetic image, a virtual non-contrast image, a virtual iodine image, a virtual pure lumen image, and an ultra-high resolution image.

17. The non-transitory computer-readable medium of claim 15, wherein combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task comprises: The results of the medical imaging analysis subtasks are combined based on statistical weighting of the results of the medical imaging analysis subtasks.

18. The non-transitory computer-readable medium of claim 15, wherein combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task comprises: The results of the medical imaging analysis subtasks are combined based on learned weights of the results of the medical imaging analysis subtasks, the learned weights being learned based on ground truth data.

19. The non-transitory computer-readable medium of claim 15, wherein combining the results of the medical imaging analysis subtasks to generate a result of the medical imaging analysis task comprises: The results of the medical imaging analysis subtasks are combined based on context-based weighting of the results of the medical imaging analysis subtasks.

20. The non-transitory computer-readable medium of claim 15, wherein the medical imaging analysis task comprises automatic reporting of coronary artery stenosis, and the plurality of medical imaging analysis subtasks comprises at least one of detection of coronary artery centerline, detection of stenosis, grading of stenosis, marking of vessel segments, and detection of lumen and plaque.

Images