Probabilistic tree pursuit and large vessel occlusion detection in medical imaging

Abstract

Systems and methods for generating a probability tree of a blood vessel are provided. An input medical image of a blood vessel of a patient is received. An anatomical landmark is identified in the input medical image. A centerline of the blood vessel in the input medical image is determined based on the anatomical landmark. A probability tree of the blood vessel is generated based on a fitted probability of the anatomical landmark and the centerline of the blood vessel. The probability tree of the blood vessel is output.

Description

Technical Field

[0001] This invention relates generally to medical imaging analysis, and particularly to probability tree tracing and large vessel occlusion detection in medical imaging. Background Technology

[0002] A stroke occurs when the blood supply to a part of the brain is interrupted or reduced, preventing brain tissue from receiving oxygen and nutrients. Timely treatment is crucial in stroke management. Current stroke protocols first determine whether the stroke is hemorrhagic or ischemic. If the stroke is ischemic, a CTA (computed tomography angiography) is performed to assess for the presence of large vessel occlusion (LVO) and / or significant blockage of major cerebral vessels.

[0003] AI-based (artificial intelligence) methods have been proposed for the automated interpretation of medical images to accelerate decision-making and support interventions in stroke protocols, thereby reducing time to treatment. However, such conventional AI-based methods for obtaining continuous vessel segmentation exhibit reduced robustness, especially in the presence of signal loss, noise, vessel tortuosity, calcification, and proximity to bone or bifurcation. Furthermore, such conventional AI-based methods for detecting LVOs cannot pinpoint the exact location of occlusions within the vascular distribution model. Summary of the Invention

[0004] According to one or more embodiments, a system and method for generating a probability tree for blood vessels are provided. The system receives an input medical image of a patient's blood vessels. Anatomical landmarks are identified in the input medical image. The centerline of the blood vessels in the input medical image is determined based on the anatomical landmarks. A probability tree for the blood vessels is generated based on the probability of fitting the anatomical landmarks and the centerline of the blood vessels. The probability tree for the blood vessels is output.

[0005] In one embodiment, the centerline of the blood vessel is determined as a centerline probability map. The probability tree is generated by: extracting centerline segments from the centerline probability map; generating candidate mappings from the centerline segments to the probability tree between the centerline segments and anatomical landmarks; calculating a score for each candidate mapping based on the detection probabilities of the anatomical landmarks and the centerline segments, as well as the position probabilities of the anatomical landmarks and the centerline segments; and selecting one of the candidate mappings based on the score.

[0006] In one embodiment, the probability tree is generated by using reinforcement learning (RL) agents to determine paths between anatomical landmarks and generating the probability tree based on the likelihood of those paths. Each of the RL agents is trained to navigate to the corresponding anatomical landmark based on geometric features and local image context features of the input medical image. One or more RL agents share geometric features and local image context features with the RL agents they are connected to. In one embodiment, the configuration of the RL agents uses a meta-agent to determine the order in which the RL agents are run.

[0007] In one embodiment, blood vessels are segmented from an input medical image based on their centerlines. A geometric index is then calculated based on the segmented vessels.

[0008] In one embodiment, the location of large vessel occlusion is detected by identifying discontinuities in the probability tree that lead to missing downstream segments.

[0009] In one embodiment, RL agents are used to identify anatomical landmarks in an input medical image. Each RL agent is trained to navigate to the corresponding anatomical landmark based on geometric features and local image context features of the input medical image. One or more RL agents share geometric features and local image context features with the RL agents they are connected to. Geometric features may include one or more of relative position, relative distance, relative angle, current agent position confidence, or embedded image intensity features.

[0010] These and other advantages of the present invention will become apparent to those skilled in the art from the following detailed description and accompanying drawings. Attached Figure Description

[0011] Figure 1 A method for generating a probability tree of blood vessels in an input medical image, according to one or more embodiments, is illustrated.

[0012] Figure 2 The workflow of an online or inference processing stage for applying one or more trained machine learning networks to generate a probability tree of blood vessels in an input medical image, according to one or more embodiments, is illustrated.

[0013] Figure 3 Exemplary anatomical landmarks are shown in images according to one or more embodiments;

[0014] Figure 4 An exemplary image of a probability tree generated according to the embodiments described herein is shown;

[0015] Figure 5Exemplary images of detected LVO (large vessel occlusion) according to one or more embodiments are shown;

[0016] Figure 6 The workflow of an offline or training processing phase for training one or more machine learning networks to generate a probability tree of blood vessels from an input medical image is illustrated according to one or more embodiments.

[0017] Figure 7 A system view of an automated stroke management system according to one or more embodiments is shown;

[0018] Figure 8 The framework of a multi-scale reinforcement learning-based model according to one or more embodiments is shown;

[0019] Figure 9 The image of a vertebra labeled using only local image context features according to the embodiments described herein is shown in comparison with an image of a vertebra labeled using a multi-scale reinforcement learning-based model using both geometric features and local image context features.

[0020] Figure 10 An image showing a probability tree generated from an input CTA (computed tomography angiography) medical image using a multi-scale reinforcement learning-based model according to embodiments described herein;

[0021] Figure 11 An exemplary artificial neural network is shown that can be used to implement one or more embodiments;

[0022] Figure 12 A convolutional neural network that can be used to implement one or more embodiments is shown; and

[0023] Figure 13 A high-level block diagram of a computer that can be used to implement one or more embodiments is shown. Detailed Implementation

[0024] This invention generally relates to methods and systems for probabilistic tree tracing and large vessel occlusion detection in medical imaging. Embodiments of the invention are described herein to provide a visual understanding of such methods and systems. Digital images typically consist of digital representations of one or more objects (or shapes). The digital representations of objects are generally described herein in terms of identifying and manipulating them. Such manipulation is a virtual manipulation performed in the memory or other circuitry / hardware of a computer system. Therefore, it is to be understood that embodiments of the invention can be performed within a computer system using data stored within the computer system.

[0025] The embodiments described herein provide probabilistic tree tracing and large vessel occlusion detection in medical imaging. The embodiments described herein use semantic knowledge of anatomical landmarks, combined with vessel centerlines identified by a trained deep learning model, to constrain and detect the semantic probabilistic representation of the vessel tree. Furthermore, the embodiments described herein use such semantic knowledge to automatically detect the presence of large vessel occlusion and compute geometric indices for interventional planning.

[0026] Figure 1 A method 100 for generating a probability tree of blood vessels in an input medical image, according to one or more embodiments, is illustrated. The steps of method 100 can be performed by one or more suitable computing devices (such as, for example...). Figure 13 The computer (1302) is used to execute this. Figure 2 A workflow 200 is illustrated, according to one or more embodiments, for an online or inference processing phase of applying one or more trained machine learning networks to generate a probability tree of blood vessels in an input medical image. The one or more trained machine learning networks are trained during a previous offline or training processing phase, as described below. Figure 6 As described in further detail. Figure 1 and Figure 2 They will be described together.

[0027] exist Figure 1 At step 102, an input medical image of the patient's blood vessels is received. In one embodiment, the input medical image is a 3D (three-dimensional) CTA (computed tomography angiography) volume. For example, the input medical image could be in... Figure 2 The input CTA volume received at step 202. However, the input medical image can belong to any other suitable modality, such as, for example, MRI (magnetic resonance imaging), ultrasound, X-ray, or any other medical imaging modality or combination of medical imaging modalities. The input medical image can include 2D (two-dimensional) images and / or 3D (three-dimensional) volumes, and can include a single input medical image or multiple input medical images. The input medical image can be received directly from an image acquisition device (such as, for example, a CT scanner) during acquisition, or it can be received by loading previously acquired medical images from the storage device or memory of a computer system (e.g., PACS (image archiving and communication system)) or by receiving medical images that have been transmitted from a remote computer system. It should be understood that references to pixels of an image herein refer equally to references to voxels of a volume (and vice versa).

[0028] exist Figure 1 In step 104, anatomical landmarks are marked in the input medical image. Figure 3Image 300 illustrates exemplary anatomical landmarks. Specifically, image 300 illustrates exemplary anatomical landmarks for the following: right middle cerebral artery, right middle cerebral artery M2, left middle cerebral artery, left middle cerebral artery M2, carotid artery confluence, right carotid frontal artery, left carotid frontal artery, basilar artery, right intracranial artery, left intracranial artery, right carotid artery entering the skull, left carotid artery entering the skull, vertebral artery confluence with basilar artery, right vertebral artery, right vertebral artery C3, right vertebral artery C5, left vertebral artery, left vertebral artery C3, left vertebral artery C5, right carotid artery bifurcation, and left carotid artery bifurcation. In one embodiment, a multi-scale reinforcement learning-based model is used to identify anatomical landmarks, the model comprising an AI (artificial intelligence) agent implemented using a graph neural network and trained using deep reinforcement learning, as described below. Figure 8 As described in further detail. However, any other suitable method can be used to identify anatomical landmarks.

[0029] In some embodiments, bones can also be removed from the input medical image. Any suitable method can be used to remove bones from the input medical image. In one embodiment, a machine learning-based depth image-to-image model can be trained to remove bones from the input medical image. In one example, in Figure 2 At step 204, bone removal and vascular landmark marking are performed on the input CTA volume.

[0030] exist Figure 1 At step 106, the centerline of the blood vessel in the input medical image is determined based on anatomical landmarks. In one embodiment, the centerline of the blood vessel is determined as a centerline probability map. The centerline probability map defines the pixel-wise probability that a corresponding pixel depicts the centerline of the blood vessel. In one example, in Figure 2 In the online processing stage 204, step 208 generates the centerline probability map.

[0031] In one embodiment, a machine learning-based model (such as, for example, an image-to-image deep learning classifier) ​​is used to determine the centerline of the blood vessel. The image-to-image deep learning classifier receives an input medical image and anatomical landmarks as input and generates a centerline probability map as output. The image-to-image deep learning classifier is trained during a previous offline or training phase, as follows: Figure 6 As described in further detail. However, any other suitable method can be used to determine the center line of the blood vessel.

[0032] exist Figure 1 In step 108, a probability tree for the blood vessel is generated based on the probabilities of fitting anatomical landmarks and the centerline of the vessel. The probability tree is a probabilistic or statistical representation of the blood vessel tree. In one embodiment, the probability tree for the blood vessel may also be generated based on the probabilities of fitting a prior probability tree.

[0033] In one embodiment, a probability tree for blood vessels is generated through combinatorial optimization. In this embodiment, centerline segments are extracted from a centerline probability map. In one example, the 3D rivulet method is applied to the centerline probability map to extract centerline segments. For example, in Figure 2 In step 208, the center line segment is extracted using the 3D rivulet method, and... Figure 2 At step 210, a probability tree is generated. In another example, the centerline segment can be extracted by running a path tracing algorithm (e.g., dynamic programming) on ​​a centerline probability map known to be connected between two anatomical landmarks. The combinatorial optimization problem is then solved by generating candidate mappings of centerline segments and anatomical landmarks to the probability tree and measuring the overall probability score of the candidates. In one embodiment, the overall probability score is calculated as the sum of the detection probabilities of the anatomical landmarks and (from the classifier) ​​segments and the probabilities of the locations of the anatomical landmarks and segments in the aligned probability tree. A penalty term is included to account for missing anatomical landmarks and / or segments. One way to calculate the overall probability score is by using a weighted linear combination of the landmark scores, the scores fitted to each centerline segment in the image, and the scores fitted to the offline-trained probabilistic vascular tree of the landmarks and centerline segments. The scores of the landmarks and centerlines fitted to the image are given by the probabilities output by each landmark and centerline network detector at the target location. The weights of the linear combination and the penalty term for missing segments are trained to have maximum values ​​under correct fitting.

[0034] In another embodiment, a probability tree for blood vessels is generated using reinforcement learning optimization. Reinforcement learning (RL) agents are trained for each corresponding segment to track paths between anatomical landmarks based on one or more of anatomical landmarks, a centerline probability map, an input medical image, or a prior probability tree of the blood vessel. The RL agent is trained to move along a path from an initial anatomical landmark to a target anatomical landmark by observing a centerline probability map as input and determining a set of actions in three dimensions (e.g., move left, right, up, down, back, forward, etc.) as output, which will move the RL agent's position closer to the target anatomical landmark. The corresponding RL agent is applied for each target blood vessel path between the vessel landmarks to determine the location of that blood vessel segment. The final tree is constructed by running all RL agents and determining the tree with the highest probability. This tree will have detected landmarks as nodes and the most probable paths taken by the RL agents (as trained) as segments between the landmarks.

[0035] In one embodiment, the meta-RL agent is trained to determine, based on current data, which RL agents should be run to optimize the overall process. The meta-RL agent observes the overall configuration of the RL agents, their current scores, and / or other local agent characteristics, and selects which RL should take over next to achieve overall convergence based on these observations. The following is about... Figure 8 This will further describe RL proxies. Other multi-proxy strategies can also be used.

[0036] exist Figure 1 At step 110, the probability tree of the blood vessels is output. For example, the probability tree of the blood vessels can be output by displaying the probability tree of the blood vessels on a display device of the computer system, storing the probability tree of the blood vessels in the memory or storage device of the computer system, or by transmitting the probability tree of the blood vessels to a remote computer system. Figure 4 It shows that according to Figure 1 An exemplary image 400 of the probability tree generated by method 100.

[0037] In one embodiment, a geometric index can be calculated for the intervention plan. For example, it can be... Figure 2 At step 212, the geometric index is calculated. In this embodiment, the blood vessel is segmented from the input medical image around its centerline. The blood vessel can be segmented from the input medical image using, for example, a deep learning segmentation classifier. The geometric index of the blood vessel is calculated based on the segmented blood vessel and its centerline. Examples of geometric indices include the length and count of bifurcations, segments, and paths; the lumen diameter of each segment; the bifurcation angle formation; and tortuosity indices such as elongation, sum of angles, or inflection point counts. Probability trees can be used to augment the geometric index with confidence intervals.

[0038] In one embodiment, LVO (Large Vessel Occlusion) can be detected in a blood vessel based on a probability tree, segments of the vessel, and the centerline of the vessel. The location of the LVO can be detected by identifying abrupt discontinuities along the probability tree that lead to the loss of a downstream segment. In another embodiment, a separate deep learning classifier can be applied to determine the probability of the LVO at the identified location. Figure 5Exemplary images 500 of detected LVOs according to one or more embodiments are shown. Image 502 shows a detected occlusion of the distal left M1 segment / proximal left M2 segment with reduced distal arborization. Image 504 shows a detected occlusion of the right MCA (middle cerebral artery) M2 branch outside the bifurcation. Image 506 shows the maximum intensity projection of the volume with occlusion on the right MCA branch (after bone removal for proper vascular visualization). Image 508 shows a systematically detected LVO in the MCA M1 branch. The location of the LVO can be confirmed by analysis of the occlusion size and composition in NCCT (non-contrast CT), DECT (delayed contrast-enhanced CT), or PCCT (photon-counting CT) scans. LVO detection according to the embodiments described herein was experimentally validated on a database of 224 datasets (64 positive and 160 hard negative – visually selected as potential LVOs), showing a sensitivity of 91% and a specificity of 69%. The detection performance of the bifurcation location of the MCA (middle cerebral artery) has a mean value of 1.5 mm for the left MCA and a mean value of 1.3 mm for the right MCA.

[0039] Figure 6 A workflow 600, according to one or more embodiments, is illustrated for an offline or training processing phase of training one or more machine learning networks to generate a probability tree of blood vessels from an input medical image. Once trained according to workflow 600, one or more trained machine learning networks can be applied during the online or inference processing phase. In one example, workflow 600 is executed to train a multi-scale reinforcement learning-based model, an image-to-image deep learning classifier, and / or a deep learning segmentation classifier. Once trained, the trained multi-scale reinforcement learning-based model can be applied to perform... Figure 1 Step 104 and Figure 2 Step 202, the trained image-to-image deep learning classifier can be applied to perform... Figure 1 Step 106 and Figure 2 Step 206, and the trained deep learning segmentation classifier can be applied to compute Figure 1 Geometric index in method 100 and execution Figure 2 Step 212.

[0040] In workflow 600, at step 602, a database of CTA training images of blood vessels is maintained. At step 604, the CTA training images are labeled using segmentation of the arterial vascular system and anatomical landmarks. In some embodiments, the centerline of the blood vessel is also directly labeled on the CTA training images. Alternatively, the centerline of the blood vessel can be extracted. At step 606, a probability tree is generated from the segmentation and anatomical landmarks, for example, using a parametric or non-parametric probability distribution model. At step 608, one or more machine learning-based networks are trained based on the probability tree and / or the labeled database of CTA training images. In one example, a multi-scale reinforcement learning-based model can be trained based on the labeled CTA training images to detect anatomical landmarks. In another example, an image-to-image deep learning classifier can be trained based on the labeled CTA training images to identify the blood vessel centerline by regressing the output of a probability distribution proportional to the distance from the centerline. In yet another example, a deep learning segmentation classifier can be trained based on the labeled CTA training images to segment blood vessels only around the centerline.

[0041] In one embodiment, the embodiments described herein can be implemented in an automated stroke management system. Figure 7 An exemplary system view 700 of an automated stroke management system according to one or more embodiments is shown. Figure 1 Method 100 can be implemented in an automated stroke management system that analyzes input medical imaging and detects probabilistic trees of LVO and intervention plans. Based on the interpretation of the imaging (directly on the imaging scanner or on any edge or computing device), automated decision-making can be performed through work list prioritization, automated notification, therapy selection (e.g., intravenous tissue plasminogen activator), and / or transferring probabilistic tree knowledge to appropriate treatments (e.g., management of endovascular thrombectomy).

[0042] Figure 8 A framework 800 for a multi-scale reinforcement learning-based model according to one or more embodiments is shown. The multi-scale reinforcement learning model can be applied to input medical images to perform… Figure 1 Step 104 and Figure 2Step 202. In the multi-scale reinforcement learning model, each RL agent is trained to navigate to the corresponding target anatomical object across images 802 of different image scales and dimensions. In contrast to traditional multi-scale deep reinforcement learning, the RL agents according to the embodiments described herein are implemented using graph neural networks to extract geometric features, along with local image context features, from the input medical image. Local image context features are features derived from the image / voxel intensities observed in the input patch. Geometric features are computed based on the xyz positions in the image, from which relative positions, distances, etc., can be derived. Exemplary geometric features include relative positions, relative distances (normalized by the global scale of the model) or relative angles, current agent position confidence, and embedded image intensity features.

[0043] Each RL agent can share features with other connected RL agents. Connected RL agents are identified by a connectivity graph 808. The connectivity graph 808 can be, for example, a user-defined predefined connectivity graph, or it can be an exhaustive connectivity graph in which each node is connected to all other nodes. Each RL agent uses extracted geometric features and local image context features, along with shared features received from other connected RL agents, to learn an action set 806 to navigate from its current position to a location closer to the target anatomical landmark using deep reinforcement learning. This is accomplished using a deep neural network 804 to learn the probabilities of the action set. For example, the action set can include moving up, moving down, moving left, moving right, moving forward, moving backward, and stopping.

[0044] The RL agents are jointly trained using geometric features and local image context features, such as deep Q-learning, policy gradients, or actor-critic models extended to multi-agent frameworks. At runtime, each RL agent starts from its intended location in the image and navigates synchronously. In one embodiment, a meta-agent is trained to coordinate the global application of the landmark RL agents. For example, the meta-agent can be trained using a state space including the global position on each landmark agent and an action space including which RL agent should move next, and / or "teleport" actions that move the landmark RL agent position to a location determined by the positions of all other agents. The meta-agent can then be trained to optimally move the entire ensemble of landmark RL agents.

[0045] In one embodiment, a single RL agent is trained on a graph neural network, where the output set of actions includes which anatomical landmark to move to next. In one embodiment, graph structure learning is performed to determine which edges of the connectivity graph are important in transferring features between RL agents. In one embodiment, the graph neural network may additionally have nodes corresponding to prior patient information, such as, for example, age, gender, etc. The transfer of such prior patient information enables the RL agent to develop search strategies that can account for changes in image appearance. In one embodiment, the graph neural network may additionally have nodes corresponding to global patient information, which can be estimated from image data and is not known prior. For example, a node corresponding to the patient's height can be added, and a posterior distribution of its value can be determined via messaging. When the node reaches stationarity, the height should also converge to a stationary distribution. In one embodiment, multi-agent training with a graph neural network is performed, where the action set includes "teleport" actions to a location estimated by the current geometric context.

[0046] In one embodiment, a multi-scale reinforcement learning-based model can be applied to spine labeling to determine vertebral labels in CT images. Individual local image context features are insufficient to resolve ambiguities between vertebral labels. Passing information to an RL agent will help resolve these ambiguities. Figure 9 A comparison of images 902 and 904 is shown. Image 902 was labeled using only local image context features. As can be seen, vertebrae T3 and T5 are mislabeled in image 902. Image 904 was labeled using a multi-scale reinforcement learning-based model with both geometric features and local image context features, according to the embodiments described herein. The vertebral labels in image 904 are accurate.

[0047] In one embodiment, a multi-scale reinforcement learning-based model can be applied to identify anatomical landmarks (e.g., in...). Figure 1 Step 104) and tracing vascular pathways between anatomical landmarks in CTA images to generate a probability tree (e.g., in Figure 1 (Step 108). Prior knowledge from the probability tree can be used to determine the connectivity of the agent associated with anatomical landmarks. Furthermore, the RL agent can be trained to track trajectories between anatomical landmarks by determining the set of actions that will cause the RL agent to follow vascular pathways. Figure 10 Image 1004 shows a probability tree generated from an input CTA medical image 1002 using a multi-scale reinforcement learning-based model according to an embodiment described herein. Prior knowledge of the probability tree in image 1006 is used to determine the connectivity of RL agents associated with anatomical landmarks.

[0048] The embodiments described herein relate to the claimed system and the claimed method. Features, advantages, or alternative embodiments described herein may be assigned to other claims, and vice versa. In other words, the system claims may be improved using features described or claimed in the context of the method. In this case, the functional characteristics of the method are embodied by the target unit providing the system.

[0049] Furthermore, some embodiments described herein relate to methods and systems for utilizing trained machine learning-based networks (or models), and to methods and systems for training machine learning-based networks. Features, advantages, or alternative embodiments described herein may be assigned to other claimed objects, and vice versa. In other words, the claims for methods and systems for training machine learning-based networks may be modified using features described or claimed in the context of methods and systems for utilizing trained machine learning-based networks, and vice versa.

[0050] Specifically, the trained machine learning-based network used in the embodiments described herein can be adapted by methods and systems for training machine learning-based networks. Furthermore, the input data of the trained machine learning-based network can include advantageous features and embodiments of the training input data, and vice versa. Similarly, the output data of the trained machine learning network can include advantageous features and embodiments of the output training data, and vice versa.

[0051] Generally, trained machine learning-based networks mimic the cognitive functions of humans in associating with other human minds. In particular, through training on training data, trained machine learning-based networks can adapt to new situations and detect and infer patterns.

[0052] Generally, the parameters of a machine learning-based network can be adapted through training. Specifically, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of a trained machine learning-based network can be iteratively adapted through several training steps.

[0053] Specifically, the trained machine learning-based network can include neural networks, support vector machines, decision trees, and / or Bayesian networks, and / or the trained machine learning-based network can be based on k-means clustering, Q-learning, genetic algorithms, and / or association rules. Specifically, the neural network can be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, the neural network can be an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0054] Figure 11 An embodiment of an artificial neural network 1100 according to one or more embodiments is shown. Alternative terms for "artificial neural network" are "neural network," "artificial neural network," or "neural network." The artificial neural network 1100 can be used to implement the machine learning networks described herein, such as those related to… Figure 1 Method 100 Figure 2 Workflow 200 Figure 6 Workflow 600 and Figure 8 The framework 800 publicly discloses machine learning-based networks.

[0055] The artificial neural network 1100 includes nodes 1102-1122 and edges 1132, 1134, ..., 1136, where each edge 1132, 1134, ..., 1136 is a directed connection from a first node 1102-1122 to a second node 1102-1122. Generally, the first node 1102-1122 and the second node 1102-1122 are different nodes 1102-1122, but it is also possible that the first node 1102-1122 and the second node 1102-1122 are the same. For example, in... Figure 11 In the diagram, edge 1132 is a directed connection from node 1102 to node 1106, and edge 1134 is a directed connection from node 1104 to node 1106. Edges 1132, 1134, ..., 1136 from the first node 1102-1122 to the second node 1102-1122 are also represented as "incoming edges" for the second node 1102-1122 and "outgoing edges" for the first node 1102-1122.

[0056] In this embodiment, nodes 1102-1122 of the artificial neural network 1100 can be arranged in layers 1124-1130, wherein the layers may include an inherent order introduced by edges 1132, 1134, ... 1136 between nodes 1102-1122. Specifically, edges 1132, 1134, ... 1136 may exist only between neighboring layers of a node. Figure 11 In the illustrated embodiment, there is an input layer 1124 consisting only of nodes 1102 and 1104 without any incoming edges, an output layer 1130 consisting only of node 1122 without any outgoing edges, and hidden layers 1126 and 1128 between the input layer 1124 and the output layer 1130. Generally, the number of hidden layers 1126 and 1128 can be chosen arbitrarily. The number of nodes 1102 and 1104 in the input layer 1124 is typically related to the number of input values ​​of the neural network 1100, and the number of nodes 1122 in the output layer 1130 is typically related to the number of output values ​​of the neural network 1100.

[0057] Specifically, (real) numbers can be assigned as values ​​to each node 1102-1122 of the neural network 1100. Here, x （n） i This represents the value of the i-th node 1102-1122 in the n-th layer 1124-1130. The values ​​of nodes 1102-1122 in the input layer 1124 are equivalent to the input values ​​of neural network 1100, and the value of node 1122 in the output layer 1130 is equivalent to the output value of neural network 1100. Furthermore, each edge 1132, 1134, ..., 1136 may include a weight as a real number, specifically a real number in the interval [-1, 1] or the interval [0, 1]. Here, w... (m,n) i,j This represents the weight of the edge between the i-th node (1102-1122) in layer m (1124-1130) and the j-th node (1102-1122) in layer n (1124-1130). Additionally, the abbreviation w... (n) i,j Defined for weight w (n,n+1) i,j .

[0058] Specifically, in order to calculate the output value of neural network 1100, the input value is propagated through the neural network. Specifically, the values ​​of nodes 1102-1122 in the (n+1)th layer 1124-1130 can be calculated based on the values ​​of nodes 1102-1122 in the nth layer 1124-1130 using the following equation:

[0059] .

[0060] In this paper, the function f is the transfer function (another term is "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logic functions, generalized logic functions, hyperbolic tangent functions, arctangent functions, error functions, smoothstep functions), or rectifier functions. Transfer functions are primarily used for normalization purposes.

[0061] Specifically, the value is propagated layer by layer through the neural network, wherein the value of the input layer 1124 is given by the input of the neural network 1100, wherein the value of the first hidden layer 1126 can be calculated based on the value of the input layer 1124 of the neural network, wherein the value of the second hidden layer 1128 can be calculated based on the value of the first hidden layer 1126, and so on.

[0062] To set the value w of the edge (m,n) i,jTraining data must be used to train the neural network 1100. Specifically, the training data includes training input data and training output data (denoted as t). i For the training step, the neural network 1100 is applied to the training input data to generate the computed output data. Specifically, the training data and the computed output data include a number of values ​​equal to the number of nodes in the output layer.

[0063] Specifically, the weights within the neural network 1100 are recursively adapted using a comparison between the calculated output data and the training data (backpropagation algorithm). Specifically, the weights are changed according to the following equation:

[0064]

[0065] Where γ is the learning rate, and if the (n+1)th layer is not the output layer, it can be based on δ. (n+1) j Recursively apply the number δ (n) j The calculation is as follows:

[0066] ,

[0067] And if the (n+1)th layer is the output layer 1130, then it is calculated as follows:

[0068]

[0069] Where f' is the first derivative of the activation function, and y (n+1) j It is the comparison training value of the j-th node in the output layer 1130.

[0070] Figure 12 A convolutional neural network 1200 according to one or more embodiments is illustrated. The convolutional neural network 1200 can be used to implement the machine learning networks described herein, such as those concerning… Figure 1 Method 100 Figure 2 Workflow 200 Figure 6 Workflow 600 and Figure 8 The framework 800 publicly discloses machine learning-based networks.

[0071] exist Figure 12In the illustrated embodiment, the convolutional neural network 1200 includes an input layer 1202, convolutional layers 1204, pooling layers 1206, fully connected layers 1208, and an output layer 1210. Alternatively, the convolutional neural network 1200 may include several convolutional layers 1204, several pooling layers 1206, and several fully connected layers 1208, as well as other types of layers. The order of the layers can be arbitrarily chosen; typically, the fully connected layer 1208 is used as the last layer before the output layer 1210.

[0072] Specifically, within the convolutional neural network 1200, the nodes 1212-1220 of a layer 1202-1210 can be considered as arranged as a d-dimensional matrix or a d-dimensional image. In particular, in the two-dimensional case, the value of the node 1212-1220 indexed by i and j in the nth layer 1202-1210 can be represented as x. (n) [i, j] However, the arrangement of nodes 1212-1220 in a layer 1202-1210 has no effect on the computation itself performed within the convolutional neural network 1200, because these are given only by the structure and weights of the edges.

[0073] Specifically, the convolutional layer 1204 is characterized by forming the structure and weights of the input edges for the convolution operation based on a certain number of kernels. In particular, the structure and weights of the input edges are chosen such that the value x of node 1214 of the convolutional layer 1204... (n) k Based on the value x of node 1212 of the previous layer 1202. (n-1) And calculated as convolution x (n) k =K k * x (n-1) In the two-dimensional case, convolution* is defined as:

[0074] .

[0075] Here, the k-th core K k This is a d-dimensional matrix (a two-dimensional matrix in this embodiment), which is typically small compared to the number of nodes 1212-1218 (e.g., a 3×3 or 5×5 matrix). Specifically, this means that the weights of the incoming edges are not independent but are chosen such that they produce the convolution equation. Specifically, for the kernel, which is a 3×3 matrix, there are only 9 independent weights (each entry in the kernel matrix corresponds to one independent weight), regardless of the number of nodes 1212-1220 in the corresponding layers 1202-1210. Specifically, for convolutional layer 1204, the number of nodes 1214 in the convolutional layer is equal to the number of nodes 1212 in the previous layer 1202 multiplied by the number of kernels.

[0076] If the nodes 1212 of the previous layer 1202 are arranged as a d-dimensional matrix, then using multiple kernels can be interpreted as adding another dimension (represented as the "depth" dimension) so that the nodes 1214 of the convolutional layer 1204 are arranged as a (d+1)-dimensional matrix. If the nodes 1212 of the previous layer 1202 have already been arranged as a (d+1)-dimensional matrix including the depth dimension, then using multiple kernels can be interpreted as extending along the depth dimension so that the nodes 1214 of the convolutional layer 1204 are also arranged as a (d+1)-dimensional matrix, where the size of the (d+1)-dimensional matrix with respect to the depth dimension is a multiple of the kernel size in the previous layer 1202.

[0077] The advantage of using convolutional layer 1204 is that it can take advantage of the spatial local correlation of the input data by enforcing a local connectivity pattern between nodes in neighboring layers, in particular by connecting each node only to a small region of nodes in the previous layer.

[0078] exist Figure 12 In the illustrated embodiment, the input layer 1202 includes 36 nodes 1212 arranged as a two-dimensional 6×6 matrix. The convolutional layer 1204 includes 72 nodes 1214 arranged as two two-dimensional 6×6 matrices, each of which is the result of convolving the values ​​of the input layer with a kernel. Equivalently, the nodes 1214 of the convolutional layer 1204 can be interpreted as arranged as a three-dimensional 6×6×2 matrix, where the last dimension is the depth dimension.

[0079] Pooling layer 1206 can be characterized by the activation function of its node 1216, which forms the pooling operation based on a nonlinear pooling function f, as well as the structure and weights of the incoming edges. For example, in the two-dimensional case, the value x of node 1216 of pooling layer 1206... (n) It can be based on the value x of node 1214 of the previous layer 1204. (n-1) And calculated as:

[0080] .

[0081] In other words, by using pooling layer 1206, the number of nodes 1214 and 1216 can be reduced by replacing a number d1·d2 of adjacent nodes 1214 in the previous layer 1204 with a single node 1216, which is calculated based on the values ​​of the number of adjacent nodes. Specifically, the pooling function f can be a maximum function, an average function, or an L2 norm function. Specifically, for pooling layer 1206, the weights of the incoming edges are fixed and are not modified through training.

[0082] The advantage of using pooling layer 1206 is that it reduces the number of nodes 1214 and 1216 and the number of parameters. This leads to a reduction in the computational cost of the network and helps control overfitting.

[0083] exist Figure 12 In the embodiment shown, pooling layer 1206 is max pooling, replacing four adjacent nodes with only one node, the maximum of the four adjacent node values. Max pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, max pooling is applied to each of the two 2D matrices, thereby reducing the number of nodes from 72 to 18.

[0084] The fully connected layer 1208 can be characterized by the fact that most, in particular all, edges exist between nodes 1216 of the previous layer 1206 and nodes 1218 of the fully connected layer 1208, and the weight of each of these edges can be adjusted individually.

[0085] In this embodiment, the nodes 1216 of the preceding layer 1206 of the fully connected layer 1208 are displayed both as a two-dimensional matrix and additionally as unrelated nodes (indicated as a row of nodes, where the number of nodes is reduced for better presentability). In this embodiment, the number of nodes 1218 in the fully connected layer 1208 is equal to the number of nodes 1216 in the preceding layer 1206. Alternatively, the number of nodes 1216 and 1218 can be different.

[0086] Furthermore, in this embodiment, the value of node 1220 in output layer 1210 is determined by applying the Softmax function to the value of node 1218 in the previous layer 1208. By applying the Softmax function, the sum of the values ​​of all nodes 1220 in output layer 1210 is 1, and all values ​​of all nodes 1220 in output layer 1210 are real numbers between 0 and 1.

[0087] The convolutional neural network 1200 may also include ReLU (rectified linear unit) layers or activation layers with nonlinear transfer functions. Specifically, the number and structure of nodes in the ReLU layer are identical to those in the previous layer. In particular, the value of each node in the ReLU layer is computed by applying the rectification function to the value of the corresponding node in the previous layer.

[0088] The inputs and outputs of different convolutional neural network blocks can be connected using summation (residual / dense neural networks), element-wise multiplication (attention), or other differentiable operators. Therefore, if the entire pipeline is differentiable, the convolutional neural network architecture can be nested rather than sequential.

[0089] Specifically, a convolutional neural network 1200 can be trained based on the backpropagation algorithm. To prevent overfitting, regularization methods can be used, such as dropout of nodes 1212-1220, random pooling, the use of artificial data, weight decay based on L1 or L2 norm, or maximum norm constraints. Different loss functions can be combined to train the same neural network to reflect the joint training objective. A subset of neural network parameters can be excluded from the optimization to preserve weights pre-trained on another dataset.

[0090] The systems, apparatus, and methods described herein can be implemented using digital circuitry or using one or more computers employing known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or be coupled to one or more mass storage devices, such as one or more disks, internal hard disks and removable disks, magneto-optical disks, optical disks, etc.

[0091] The systems, apparatus, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such systems, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0092] The systems, apparatuses, and methods described herein can be implemented within a network-based cloud computing system. In such a system, a server or other processor connected to the network communicates with one or more client computers via the network. Client computers can communicate with the server via, for example, a web browser application residing and operating on the client computer. Client computers can store data on the server and access that data via the network. Client computers can transmit requests for data or for online services to the server via the network. The server can perform the requested service and provide data to (multiple) client computers. The server can also transmit data suitable for causing client computers to perform specified functions (e.g., perform calculations, display specified data on a screen, etc.). For example, the server can transmit data suitable for causing client computers to perform one or more steps or functions of the methods and workflows described herein (including...). Figure 1 , 2 A request for one or more steps or functions of the methods and workflows described herein (including steps or functions of step 6). Figure 1 , 2One or more steps or functions (or steps 6) may be performed by a server or by another processor in a network-based cloud computing system. Some steps or functions of the methods and workflows described herein (including...) Figure 1 , 2 (or one or more steps of step 6) can be performed by a client computer in a web-based cloud computing system. The steps or functions of the methods and workflows described herein (including...) Figure 1 , 2 (or one or more steps of step 6) can be performed by the server and / or by client computers in a web-based cloud computing system in any combination.

[0093] The systems, apparatuses, and methods described herein can be implemented using a computer program product tangibly contained in an information carrier (e.g., contained in a non-transitory machine-readable storage device) for execution by a programmable processor; and the methods and workflow steps described herein (including Figure 1 , 2 (or one or more steps or functions of 6) can be implemented using one or more computer programs executable by such processors. A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform an activity or produce a result. A computer program can be written in any form of programming language (including compiled or interpreted languages) and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0094] Figure 13 A high-level block diagram of an example computer 1302, which can be used to implement the systems, apparatus, and methods described herein, is depicted. Computer 1302 includes a processor 1304 operatively coupled to a data storage device 1312 and a memory 1310. Processor 1304 controls such operations by executing computer program instructions that define the overall operation of computer 1302. The computer program instructions may be stored in data storage device 1312 or other computer-readable medium and loaded into memory 1310 when execution of the computer program instructions is desired. Therefore, Figure 1 , 2 The methods and workflow steps or functions of 6 may be defined by computer program instructions stored in memory 1310 and / or data storage device 1312, and controlled by processor 1304 that executes the computer program instructions. For example, the computer program instructions may be implemented to be programmed by those skilled in the art to perform... Figure 1 , 2 Or, 6. Computer-executable code outlining the methods, workflow steps, or functions. Therefore, by executing computer program instructions, processor 1304 performs... Figure 1, 2 Alternatively, methods and workflow steps or functions may be included. Computer 1302 may also include one or more network interfaces 1306 for communicating with other devices via a network. Computer 1302 may also include one or more input / output devices 1308 (e.g., monitor, keyboard, mouse, speaker, buttons, etc.) for enabling user interaction with computer 1302.

[0095] Processor 1304 may include both general-purpose microprocessors and special-purpose microprocessors, and may be the sole processor of computer 1302 or one of multiple processors. For example, processor 1304 may include one or more central processing units (CPUs). Processor 1304, data storage device 1312 and / or memory 1310 may include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs), supplemented by or incorporated therein.

[0096] Each of the data storage device 1312 and the memory 1310 includes a tangible, non-transitory computer-readable storage medium. Each of the data storage device 1312 and the memory 1310 may include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate synchronous dynamic random access memory (DDRRAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more disk storage devices (e.g., internal hard disks and removable disks), magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices (e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital versatile disc read-only memory (DVD-ROM) discs), or other non-volatile solid-state memory devices.

[0097] Input / output device 1308 may include peripheral devices such as printers, scanners, displays, etc. For example, input / output device 1308 may include display devices (such as cathode ray tube (CRT) or liquid crystal display (LCD) monitors) for displaying information to a user, a keyboard, and pointing devices such as mice or trackballs through which the user can provide input to computer 1302.

[0098] Image acquisition device 1314 can be connected to computer 1302 to input image data (e.g., medical images) into computer 1302. It is possible to implement image acquisition device 1314 and computer 1302 as a single device. It is also possible for image acquisition device 1314 and computer 1302 to communicate wirelessly via a network. In a possible embodiment, computer 1302 may be remotely located relative to image acquisition device 1314.

[0099] Any or all of the systems and apparatus discussed herein may be implemented using one or more computers (such as computer 1302).

[0100] Those skilled in the art will recognize that actual computer or computer system implementations may have other structures and may include other components, and for illustrative purposes, Figure 13 It is a high-level representation of some components of this type of computer.

[0101] The foregoing specific embodiments are to be understood in each aspect as illustrative and exemplary, not restrictive, and the scope of the invention disclosed herein is not determined by these specific embodiments, but rather more precisely by the claims as interpreted under the full breadth permitted by patent law. It is to be understood that the embodiments shown and described herein merely illustrate the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention. Various other combinations of features can be implemented by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A method comprising: Receives input medical images of the patient's blood vessels; Identify anatomical landmarks in the input medical images; The centerline of the blood vessel in the input medical image is determined based on the anatomical landmarks; Generating a probability tree for the blood vessels based on the fitting probabilities of the anatomical landmarks and the centerlines of the blood vessels includes using reinforcement learning (RL) agents to determine paths between the anatomical landmarks and generating the probability tree based on the likelihood of the paths. Each of the RL agents is trained to navigate to a corresponding anatomical landmark based on geometric features and local image context features of the input medical image, and one or more of the RL agents share the geometric features and local image context features with connected RL agents. Output the probability tree of the blood vessels.

2. The method of claim 1, wherein determining the centerline of the blood vessel in the input medical image based on the anatomical landmarks comprises: Generating a probability map of the centerline of the blood vessel, wherein generating a probability tree of the blood vessel based on the fitted probabilities of the anatomical landmarks and the centerline of the blood vessel includes: Extract the centerline segment from the centerline probability map; Generate candidate mappings from the center line segment and the anatomical landmarks to the probability tree; A score for each candidate mapping is calculated based on the detection probability of the anatomical landmarks and the centerline segment, as well as the positional probability of the anatomical landmarks and the centerline segment; and One of the candidate mappings is selected based on the score.

3. The method according to claim 1, further comprising: The configuration of the RL agent is used to determine the order in which the RL agents are run.

4. The method according to claim 1, further comprising: The blood vessels are segmented from the input medical image based on their centerline; as well as The geometric index of the blood vessel is calculated based on the segmented blood vessel.

5. The method of claim 1, further comprising: The location of large vessel occlusion is detected by identifying discontinuities in the probability tree that lead to missing downstream segments.

6. The method of claim 1, wherein identifying anatomical landmarks in the input medical image comprises: Reinforcement learning (RL) agents are used to identify the anatomical landmarks in the input medical image. Each of the RL agents is trained to navigate to the corresponding anatomical landmark based on the geometric features and local image context features of the input medical image, and one or more of the RL agents share the geometric features and the local image context features with the RL agents they are connected to.

7. The method of claim 6, wherein the geometric features include one or more of relative position, relative distance, relative angle, current agent position confidence, or embedded image intensity features.

8. An apparatus comprising: A device for receiving input medical images of a patient's blood vessels; Device for identifying anatomical landmarks in the input medical image; A device for determining the centerline of the blood vessel in the input medical image based on the anatomical landmarks; An apparatus for generating a probability tree of the blood vessels based on the probability of fitting the anatomical landmarks and the centerline of the blood vessels, comprising means for determining paths between the anatomical landmarks using reinforcement learning (RL) agents, and means for generating the probability tree based on the likelihood of the paths, wherein each of the RL agents is trained to navigate to the corresponding anatomical landmark based on geometric features and local image context features of the input medical image, and wherein one or more of the RL agents share the geometric features and the local image context features with connected RL agents; as well as A device for outputting a probability tree of the blood vessels.

9. The apparatus of claim 8, wherein the means for determining the centerline of the blood vessel in the input medical image based on the anatomical landmarks comprises: A probability map of the centerline of the blood vessel is generated, and the means for generating a probability tree of the blood vessel based on the probability of fitting the anatomical landmarks and the centerline of the blood vessel includes: Apparatus for extracting centerline segments from the centerline probability map; A means for generating candidate mappings to the probability tree between the centerline segment and anatomical landmarks; A means for calculating a score for each candidate mapping based on the detection probabilities of the anatomical landmarks and the centerline segment, and the positional probabilities of the anatomical landmarks and the centerline segment; and A means for selecting one of the candidate maps based on the score.

10. The device according to claim 8, further comprising: A means for using meta-agents to determine the order in which the RL agents are run based on the configuration of the RL agents.

11. A non-transitory computer-readable medium storing computer program instructions, which, when executed by a processor, cause the processor to perform operations including: Receives input medical images of the patient's blood vessels; Identify anatomical landmarks in the input medical images; The centerline of the blood vessel in the input medical image is determined based on the anatomical landmarks; Generating a probability tree for the blood vessels based on the fitting probabilities of the anatomical landmarks and the centerlines of the blood vessels includes using reinforcement learning (RL) agents to determine paths between the anatomical landmarks and generating the probability tree based on the likelihood of the paths. Each of the RL agents is trained to navigate to a corresponding anatomical landmark based on geometric features and local image context features of the input medical image, and one or more of the RL agents share the geometric features and local image context features with connected RL agents. Output the probability tree of the blood vessels.

12. The non-transitory computer-readable medium of claim 11, wherein determining the centerline of the blood vessel in the input medical image based on the anatomical landmarks comprises: Generating a probability map of the centerline of the blood vessel, wherein generating a probability tree of the blood vessel based on the fitted probabilities of the anatomical landmarks and the centerline of the blood vessel includes: Extract the centerline segment from the centerline probability map; Generate candidate mappings from the center line segment and the anatomical landmarks to the probability tree; A score for each candidate mapping is calculated based on the detection probabilities of the anatomical landmarks and the centerline segment, as well as the positional probabilities of the anatomical landmarks and the centerline segment; and One of the candidate mappings is selected based on the score.

13. The non-transitory computer-readable medium of claim 11, wherein the operation further comprises: Segmenting the blood vessels from the input medical image based on their centerline; and The geometric index of the blood vessel is calculated based on the segmented blood vessel.

14. The non-transitory computer-readable medium of claim 11, wherein the operation further comprises: The location of large vessel occlusion is detected by identifying discontinuities in the probability tree that lead to missing downstream segments.

15. The non-transitory computer-readable medium of claim 11, wherein identifying anatomical landmarks in the input medical image comprises: Reinforcement learning (RL) agents are used to identify the anatomical landmarks in the input medical image. Each of the RL agents is trained to navigate to the corresponding anatomical landmark based on the geometric features and local image context features of the input medical image, and one or more of the RL agents share the geometric features and the local image context features with the RL agents they are connected to.

16. The non-transitory computer-readable medium of claim 15, wherein the geometric features include one or more of relative position, relative distance, relative angle, current agent position confidence, or embedded image intensity features.

Images