Computer-implemented method, system, and storage medium for predicting a disease quantification parameter of an anatomical structure

Abstract

The application discloses a computer-implemented method, system and storage medium for predicting a disease quantification parameter of an anatomical structure. The method comprises: extracting a centerline structure based on a medical image; predicting a disease quantification parameter of each sampling point on the extracted centerline structure by using a GNN, wherein each node of the GNN corresponds to a sampling point on the extracted centerline structure, and each edge of the GNN corresponds to a spatial constraint relationship between the sampling points; for each node, extracting local features based on an image block of the corresponding sampling point by using a local feature encoder; extracting global features based on a group of image blocks of a group of sampling points, which includes the corresponding sampling point and has a spatial constraint relationship defined by the centerline structure, by using a global feature encoder; obtaining embedding features based on both the local features and the global features, and inputting the embedding features into the node. The method can integrate local and global considerations of the sampling points into the GNN, thereby improving the prediction accuracy.

Description

[0001] Cross-referencing

[0002] This application is based on and claims priority to U.S. Provisional Application No. 63 / 178,940, filed April 23, 2021, which is incorporated herein by reference. Technical Field

[0003] This invention relates to the field of anatomical structure analysis using artificial intelligence, and particularly to a computer-implemented method, system, and storage medium for predicting disease-quantitative parameters of anatomical structures. Background Technology

[0004] Accurate disease quantification parameters of anatomical structures contribute to precise diagnosis. For example, fractional flow reserve (FFR) has proven to be a reliable indicator for assessing myocardial ischemia. FFR can be measured via a pressure guidewire, but this procedure is invasive and only one or a few values ​​are measured across the entire coronary tree. Learning-based methods have been attempted to estimate FFR. This learning-based FFR estimation is essentially a low-data problem because ground truth measurements are provided only at one, a few, or even just a handful of locations. Providing accurate predictions across the entire coronary tree is extremely challenging due to the limited number of invasive FFR values ​​(measured via a pressure guidewire) available for the training process. Existing machine learning-based methods require simulated FFR values ​​as ground truth for training the model. However, simulated FFR values ​​are typically calculated using numerical flow simulations, which are both time-consuming and inaccurate for training machine learning models. Therefore, the performance of machine learning-based methods is highly limited by simulation methods, resulting in poor predictive performance and low accuracy in predicting disease quantification parameters of anatomical structures. Summary of the Invention

[0005] This invention provides a computer-implemented method for predicting disease quantification parameters of anatomical structures. It can seamlessly integrate information from sampling points of the central structure throughout the entire anatomical structure. By integrating local and global considerations of the sampling points into a graph neural network (GNN), it can accurately predict disease quantification parameters of anatomical structures using only limited labeled data.

[0006] To address the aforementioned technical problems, the embodiments of the present invention employ the following technical solutions.

[0007] According to a first aspect of the present invention, a computer-implemented method for predicting disease quantification parameters of anatomical structures is provided. The method may include: acquiring a medical image containing anatomical structures; extracting a centerline structure based on the medical image; predicting disease quantification parameters for each sampling point on the extracted centerline structure by utilizing a graph neural network (GNN), where each node of the GNN corresponds to a sampling point on the extracted centerline structure, and each edge of the GNN corresponds to a spatial constraint relationship between two sampling points; extracting local features based on image patches corresponding to the sampling points using a local feature encoder; extracting global features based on a set of image patches containing a set of sampling points using a global feature encoder, wherein the set of sampling points includes the corresponding sampling points and has a spatial constraint relationship defined by the centerline structure; obtaining embedded features based on both the local features and the global features, and inputting the embedded features into the node of the GNN.

[0008] According to a second aspect of the present invention, a system for predicting disease quantification parameters of anatomical structures is provided, the system comprising: an interface configured to acquire medical images containing anatomical structures; and a processor configured to execute methods for predicting disease quantification parameters of anatomical structures according to various embodiments of the invention.

[0009] According to a third aspect of the present invention, the present invention provides a non-transitory computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, implement the method for predicting disease quantification parameters of anatomical structures according to various embodiments of the present invention.

[0010] The beneficial effects of the embodiments of the present invention are as follows:

[0011] This method extracts local features from sampling points of the central axis structure and extracts global features based on a set of sampling points. It uses local and global features to obtain embedding features for input into the GNN, enabling seamless integration of information from sampling points of the central axis structure throughout the entire anatomical structure. By integrating local and global considerations of sampling points into the GNN, it can accurately predict disease quantification parameters of anatomical structures using only limited label data. Attached Figure Description

[0012] Figure 1(a) shows a flowchart of a method for predicting disease quantification parameters of anatomical structures according to an embodiment of the present invention;

[0013] Figure 1(b) shows a flowchart of the method used in Figure 1(a) according to an embodiment of the present invention;

[0014] Figure 2(a) shows a flowchart of an exemplary method for predicting disease quantification parameters of anatomical structures according to an embodiment of the present invention;

[0015] Figure 2(b) shows a flowchart of an exemplary method for local feature learning according to an embodiment of the present invention;

[0016] Figure 2(c) shows a flowchart of an exemplary method for global feature learning according to an embodiment of the present invention;

[0017] Figure 2(d) shows a flowchart of an exemplary method for predicting disease quantification parameters of anatomical structures using local and global features according to an embodiment of the present invention;

[0018] Figure 3 This paper illustrates a framework for an exemplary method of using graph neural networks to train graph neural networks for anatomical structures and predict disease quantitative parameters according to embodiments of the present invention.

[0019] Figure 4 A schematic diagram of the framework of a system for predicting disease quantification parameters of anatomical structures according to an embodiment of the present invention is shown. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but this is not intended to limit the present invention.

[0021] This invention proposes a method for predicting disease quantification parameters of anatomical structures. Figure 1(a) shows a flowchart of the method for predicting disease quantification parameters of anatomical structures according to an embodiment of the present invention. As shown in Figure 1(a), the method begins with step S101, acquiring a medical image containing anatomical structures.

[0022] It is understood that the term "anatomical structure" in this document can refer to blood vessels, airways, etc., which have a tree-like structure, but the present invention is not limited thereto. "Medical image" can refer to a complete image or an image patch cropped from a complete image, and can be in two-dimensional (2D) or three-dimensional (3D) form. It is understood that medical images can be obtained using various medical devices, such as CT images, MRI images, ultrasound images, etc., for example, MRI images containing blood vessels, MRI images of brain structures, etc.

[0023] In S102, the centerline structure is extracted based on medical images.

[0024] In some embodiments, neural networks can be used to extract the central axis structure. A neural network algorithm is a mathematical model that mimics the behavior of the brain's neural networks, performing distributed parallel information processing. This type of network, depending on the complexity of the system, adjusts the connections between its internal neurons to process information. Using a trained neural network to process medical images containing anatomical structures can quickly and accurately extract the central axis structure of the anatomical structures.

[0025] In some embodiments, a neural network can be used to segment medical images containing anatomical structures to extract the centerline structure. Subsequently, the centerline structure can be sampled to obtain sampling points. For example, the sampling points can be locations where disease quantification parameters are of interest. For instance, a set of points can be selected from points along the distribution of blood vessels as sampling points to obtain embedded features and predict the distribution of disease quantification parameters (e.g., FFR) of blood vessels.

[0026] Features can then be extracted from these sampling points as vertices (nodes) of the graph to create a graphical representation. In particular, these features can be disease-related features.

[0027] In some embodiments, the segmentation of the centerline structure of the anatomical structure can be obtained automatically, semi-automatically, or manually. Next, points on the centerline are sampled as vertices (V) of the graph (G). For each sampled point on the centerline, feature information (e.g., a local feature map) can be extracted based on image patches (or mask patches) at each sampled point. For a set of sampled points on the centerline, feature information can also be extracted based on a set of image patches at the corresponding set of sampled points (e.g., a global feature map). The following detailed explanation is primarily based on this scenario. However, it should be understood that the basis for feature information extraction is not limited to image patches at each (set of) sampled points, but can also include various predefined disease-related features at each (set of) sampled points. Specifically, disease-related features can include, but are not limited to, structural features, intensity features, and / or other derived features. As examples of structural features, geometric features can include any of the following: radius, area, narrowing, volume, length, curvature, etc. Intensity features can include any of the intensity-related measurements, such as intensity statistics (minimum, maximum, average, etc.). Other derived features can be any features derived from tree structure, intensity, or even information related to other anatomical structures. In some embodiments, disease-related features may also include pressure drop or resistance estimated using simplified equations. As input, other channels can be stacked to represent various disease-related features, such as, but not limited to, geometric feature information, intensity feature information, blood feature information, etc.

[0028] Points on the centerline structure can be linked by edges, which can be undirected. In some embodiments, edges can also be directed. In particular, an undirected edge can be considered as two directed edges, indicating a relationship or association between two nodes. Generally, directed edges can carry more information than undirected edges.

[0029] In some embodiments, information can propagate from the root to the tip of an anatomical structure, and it can also propagate in the opposite direction (e.g., from the tip of a tree to the root). In other words, information propagation or transmission between nodes of an anatomical structure can be achieved by considering both directions.

[0030] According to the present invention, taking the anatomical structure as a vascular tree structure as an example, the tree T is associated with the graph GT = (V, E), where the node vi ∈ V corresponds to the feature vector or embedding of the point on the central line structure (both have ground truth and unknown values), and the edge ei ∈ E corresponds to the directed or undirected edge between the points. According to this disclosure, both implicit representation (i.e., feature embedding) and explicit relations (i.e., graph) are fused to learn a disease prediction model for the entire anatomical structure.

[0031] In S103, a graph neural network (GNN) is constructed, such that each node of the GNN corresponds to a sampling point on the extracted centerline structure, and each edge of the GNN corresponds to the spatial constraint relationship between the sampling points. The GNN constructed in this way can learn to pass information (message passing) between nodes with ground truth values ​​and nodes without ground truth values ​​through training, achieving good training results even when labeled data is scarce.

[0032] Specifically, each node of the GNN is set as a sampling point corresponding to the centerline structure, and each edge of the GNN corresponds to the spatial constraint relationship between the sampling points. The spatial constraint relationship between sampling points can reflect the relative relationship between each sampling point, such as the direction of information transmission between sampling points. By setting each edge of the GNN to correspond to the spatial constraint relationship between sampling points, the GNN can consider the relative relationship between sampling points when predicting the disease quantification parameters of anatomical structures, thereby obtaining more accurate prediction results. It is not necessary to consider that the GNN must be physically formed and stored as a whole as a graph. Instead, the GNN can be formed and stored only by identifying nodes and edges.

[0033] In S104, the disease quantification parameters of each sampling point on the extracted centerline structure are predicted using GNN. Specifically, for each node of GNN, local features are extracted based on the image patch of the corresponding sampling point using a local feature encoder.

[0034] In S105, a set of image blocks based on a set of sampling points is used to extract global features by using a global feature encoder. The set of sampling points includes corresponding sampling points and has spatial constraint relationships defined by the centerline structure.

[0035] Specifically, 2D / 3D image patches can be extracted from sampling points on each centerline structure. Then, local features corresponding to each sampling point can be extracted using a local feature encoder. These local features can reflect the disease-related characteristics of that sampling point.

[0036] For a set of sampling points, there are spatial constraints defined by the centerline structure. These spatial constraints reflect the relative relationships between the sampling points in the set and the overall feature information of the set. For example, the overall information transmission direction and feature information change trend of the set of sampling points. In S105, global features are extracted from a set of image patches of a set of sampling points to obtain the overall feature information of each set of sampling points.

[0037] In some embodiments, the global feature encoder and the local feature encoder may be the same encoder or different encoders, as long as they can obtain the local features of the corresponding sampling point and the global features of a set of sampling points. This invention does not impose specific limitations on this.

[0038] In S106, embedding features are obtained based on both local and global features, and these embedding features are then input into the GNN node. Specifically, embedding features are obtained using local features that reflect the disease-related characteristics of each sampling point and global features that reflect the overall feature information of a group of sampling points. These embedding features are then input into the GNN to predict the disease quantification parameters of anatomical structures. Thus, in the process of using GNN to predict the disease quantification parameters of anatomical structures, not only the local features of each sampling point are considered, but also the overall features of a group of sampling points. This effectively utilizes the GNN's learning ability to transfer information between nodes with ground truth values ​​and nodes without ground truth values, thereby achieving more accurate prediction results.

[0039] It should be noted that FFR is a reliable indicator for assessing myocardial ischemia. This article uses the prediction of FFR as an example to describe the technical solution of the present invention, but the present invention is not limited thereto. Rather, the present invention is applicable to the quantitative parameters of any disease in any anatomical structure.

[0040] Taking FFR prediction as an example, the medical image can be a medical image containing blood vessels, and the anatomical structure can be a vascular tree structure. The centerline structure of the vascular tree can be extracted based on the medical image containing blood vessels.

[0041] In some embodiments, for medical images containing blood vessels, an initial arterial segmentation with a centerline structure is generated, which can be obtained automatically, semi-automatically, or manually. For each sampling point on the centerline structure, disease-related features can be extracted, which may include, but are not limited to, structural features, intensity features, and / or other derived features. For predicting FFR, these disease-related features may be pressure drop or resistance estimated using simplified equations.

[0042] Then, after extracting the centerline structure of the vascular tree, a graph neural network (GNN) is constructed, such that each node of the GNN corresponds to a sampling point on the extracted centerline structure of the vascular tree, and each edge of the GNN corresponds to the spatial constraint relationship between the sampling points. The input to each node in the GNN can be a 2D / 3D image patch of the corresponding centerline structure sampling point, and the output of each node is a quantification parameter of the vascular disease. Furthermore, the method may include obtaining a set of labeled node data, where the number of nodes in this set is less than the total number of nodes in the GNN. Further, the method may also include training the GNN based on the labeled node data by passing information between this set of nodes and other nodes.

[0043] A set of image patches based on a set of sampling points is used to extract global features using a global feature encoder. Embedded features are obtained based on both local and global features, and the embedded features are input into the nodes of a GNN. The GNN is then used to predict the FFR distribution at a series of sampling points.

[0044] In some embodiments, a GNN may include a Graph Convolutional Neural Network (GCN). The goal of a GCN is to extend the convolutional neural network architecture to non-Euclidean domains (e.g., graphs). Specifically, graph convolutions can be defined directly on the graph, allowing a series of convolutional operations to be performed for each node using graph convolutional layers and considering its spatially close neighbors. In this way, not only can local hidden information of a node be extracted, but also physical constraints in the non-Euclidean domain between the node and its surrounding nodes can be embedded, resulting in better training results and more accurate predictions even when ground truth is lacking.

[0045] The method for predicting disease quantification parameters of anatomical structures provided by this invention extracts local features from sampling points of the centerline and extracts global features based on a set of sampling points. It uses local and global features to obtain embedding features for input into a GNN, enabling seamless integration of information from sampling points of the centerline structure throughout the entire anatomical structure. By integrating local and global considerations of the sampling points into the GNN, it is possible to accurately predict disease quantification parameters of anatomical structures using only limited labeled data.

[0046] In some embodiments, each of the local feature encoder and the global feature encoder employs a two-level attention mechanism by utilizing a first-level attention block and a second-level attention block, wherein the first-level attention block is configured to extract a lesion location-aware attention map, and the second-level attention block is configured to extract a pixel-to-pixel interdependent attention map.

[0047] Understandably, attention mechanisms can consider the degree of neighboring nodes (also known as nearest neighbors) and equilateral information such as attention weights, integrating the relationships between various information and allowing the model to dynamically focus on specific parts of the input to more effectively complete the current task. A two-level attention mechanism is constructed using a first-level attention block for extracting the lesion location-aware attention map and a second-level attention block for extracting pixel-to-pixel interdependent attention maps. This two-level attention mechanism serves as a local feature encoder and a global feature encoder, enabling each encoder to consider not only the interdependent attention relationship between the lesion location and the extracted pixels, but also the correlation between the lesion location and the interdependent attention relationship between pixels. This allows each encoder to extract richer feature information from the image patch of the corresponding sampling point, thereby achieving higher accuracy when using GNN to predict the quantification parameters of anatomical structures.

[0048] Taking FFR prediction as an example, the input medical image can be a medical image containing blood vessels, such as an MRI image containing blood vessels. As shown in Figure 1(b), the input medical image 101 is processed. After extracting the centerline structure 102 of the vascular tree from the medical image 101, image blocks / mask blocks 103 are extracted from the sampling points on the centerline structure. The extracted image blocks / mask blocks 103 are then input into a global-local dual-level attention high-order GNN network 104, and the predicted result 105 is output. The first-level attention block of the global-local dual-level attention high-order GNN network 104 can extract a stenosis-aware attention map of the blood vessel. The second-level attention block can employ a non-local network to detect a more accurate pixel-to-pixel interdependent attention map. The first-level and second-level attention blocks provide more discriminative feature representations for disease prediction and lesion localization (e.g., stenosis localization in FFR prediction), resulting in higher accuracy in FFR prediction.

[0049] As shown in Figure 1(b), the GNN can include a higher-order GNN. Specifically, in a typical GNN, nodes only receive latent representations from their immediate (first-level) nearest neighbors, while higher-order GNNs utilize higher-order information passing, allowing nodes to receive latent representations from their immediate (first-level) nearest neighbors and from neighbors at levels N in each information passing step. Employing higher-order GNNs allows for the incorporation of more higher-order information, enabling training from only one or a few labeled data points and propagating information from labeled to unlabeled data points. This results in more accurate predictions when forecasting disease quantification parameters of anatomical structures.

[0050] In some embodiments, the anatomical structure includes an anatomical tree structure or an anatomical diagram structure. For example, the coronary arteries are a tree-like structure, and the method proposed in this invention can be well used for non-invasive functional assessment of coronary artery fractional flow reserve (FFR) or detection of coronary artery stenosis / lesions, obtaining more accurate prediction results.

[0051] In some embodiments, the local feature encoder includes: a first convolutional neural network (CNN) incorporating an attention mechanism for localization, and a first self-attention block following thereto, the first self-attention block incorporating an attention mechanism for inter-pixel correlation. Figure 2(a) shows a flowchart of a method for predicting disease quantification parameters of anatomical structures according to a specific embodiment of the present invention. As shown in Figure 2(a), for a 2D / 3D image block 200 on the centerline, a local feature encoder 201 is used to obtain local features 204. Specifically, the CNN incorporating an attention mechanism for localization (i.e., the 2D / 3D CNN with attention in Figure 2(a) (first-level attention) 202) is combined with the first self-attention block (i.e., the self-attention block (second-level attention) 203 in Figure 2(a)), and the local features 204 are output using the first self-attention block (i.e., the self-attention block (second-level attention) 203 in Figure 2(a) incorporating an attention mechanism for inter-pixel correlation).

[0052] Figure 2(b) illustrates a specific embodiment in which a local feature encoder 201 is used to acquire local features 204 based on a 2D / 3D image patch 200 along a centerline. For example, the local feature encoder 201 may sequentially include a 2D / 3D CNN 202 with first-level attention and a self-attention block 203.

[0053] In some embodiments, the first convolutional neural network (CNN) can be any 2D / 3D CNN network, such as VGG, ResNet, etc. It is understood that the first convolutional neural network can be any attention mechanism used for localization, such as residual attention, and the first self-attention block can be any pixel-to-pixel correlation method, such as a nonlocal network, etc. This invention does not specifically limit these methods.

[0054] In some embodiments, the global feature encoder includes: a second convolutional neural network (CNN) incorporating an attention mechanism for localization, a recurrent neural network (RNN) following it, and a second self-attention block incorporating an attention mechanism for inter-pixel correlation, wherein the RNN is configured to accumulate information from the root of the anatomical structure to the current sampling point. Specifically, as shown in FIG2(a), for the global feature encoder 205, a 2D / 3D CNN with attention (first-level attention) 206 is combined with an RNN 207, and then the output of the RNN 207 is passed to a self-attention block (second-level attention) 208, which is then used to extract global features 209. Specifically, RNN 207 is widely applicable to sequence learning tasks because it can capture both long-term and short-term information. Using RNN 207 to learn global features, for example in the FFR prediction problem, RNN 207 is configured to accumulate information from the root of the coronary artery to the current point. This allows for better acquisition of global features of the centerline structure, resulting in higher accuracy of disease quantification parameters of the anatomical structure predicted based on the acquired global features. Figure 2(c) illustrates a specific embodiment using a global feature encoder 205 to acquire global features 209 based on 2D / 3D image patches 200 along the centerline. For example, the global feature encoder 205 may sequentially include a 2D / 3D CNN 206 with first-level attention, an RNN 207, and a self-attention block 208.

[0055] As shown in Figure 2(d), after obtaining the local features 204 obtained using Figure 2(b) and the global features 209 obtained using Figure 2(c), the local features 204 and the global features 209 are connected (connection 210 as shown in Figure 2(d)) to obtain embedded features. Then, high-order GNN feature embedding 211 is performed, and the embedded features are input to each node 212 to obtain the prediction results 213 of the disease quantification parameters of the anatomical structure.

[0056] In some embodiments, the second convolutional neural network (CNN) can be any 2D / 3D CNN network, such as VGG, ResNet, etc. The second CNN can be any attention mechanism used for localization, such as residual attention. The RNN can use any RNN unit, such as LSTM, GRU, convLSTM, convGRU, etc. The second self-attention block can be any pixel-to-pixel correlation method, such as nonlocal networks, etc. This invention does not impose any specific limitations in these aspects.

[0057] It is understood that the attention mechanism used for localization in the first convolutional neural network and the second convolutional neural network can be the same attention mechanism, such as the 2D / 3D CNN with attention shown in Figure 2(a), or different attention mechanisms can be used. This invention does not specifically limit this.

[0058] It is understood that the first self-attention block and the second self-attention block can be the same or different, and the present invention does not make specific limitations in this regard.

[0059] In some embodiments, the attention mechanism used for localization includes residual attention, and each self-attention block includes a nonlocal network. For predicting FFR, the first-level attention block employing residual attention can extract a narrow-location-aware attention map, while the self-attention block employing a nonlocal network can detect a more accurate pixel-to-pixel interdependent attention map, providing a more discriminative feature representation for predicting FFR scores and narrow-location localization, resulting in higher prediction accuracy.

[0060] In some embodiments, obtaining the embedded features based on both local and global features further includes concatenating the local and global features to obtain the embedded features (as shown in Figure 2(a)). Specifically, the extracted local features can be expanded into a one-dimensional vector, and the extracted global features can also be expanded into a one-dimensional vector. Then, the two expanded one-dimensional vectors are concatenated to obtain the embedded features, so that the embedded features contain both local and global features. Inputting these embedded features into a GNN can yield more accurate prediction results.

[0061] In some embodiments, the GNN includes a graph convolution-based neural network or a graph gating-based neural network.

[0062] Specifically, graph convolutional neural networks (GCNs) can extend the CNN architecture to non-Euclidean domains. Graph convolution can be defined directly on a graph, operating on spatial nearest neighbors. A graph convolutional neural network can be represented as: Z = GCN(X, A), where X ∈ R. N×C The input is Z, where N is the number of nodes, C is the dimension of the feature embedding, A is the adjacency matrix to indicate whether there are edges between nodes (where A is determined by the centerline structure), and Z is the output of the graph convolutional neural network. It is understood that other commonly used methods in CNNs can also be used in graph convolutional neural networks, such as skip connections or attention, and this invention does not specifically limit these methods.

[0063] For example, graph-gated neural networks such as GRU or LSTM can be used in the propagation step to improve the long-term propagation of information in the graph structure. For instance, if the edges of the graph are directed, the parent node can selectively combine information from each child node by using a gating mechanism. More specifically, each graph unit (which can be a GRU or LSTM unit) contains input and output gates, a memory unit, and a hidden state. Instead of a single forget gate, each graph unit contains a forget gate for each child. Message passing can be bottom-up, top-down, or bidirectional. It is understood that the graph unit can be any RNN unit, such as LSTM, GRU, CLSTM, CGRU, etc., and this invention does not specifically limit it.

[0064] In some embodiments, the anatomical structure includes vascular structures or airway structures. The method provided by this invention can predict disease quantification parameters of such anatomical tree structures based on embedded features formed from local and global features of vascular or airway structures, and GNNs. It can seamlessly integrate information from sampling points of the central structure throughout the entire anatomical structure, and by considering both local and global factors of the sampling points, it can accurately predict disease quantification parameters of the anatomical tree structure using only limited labeled data (e.g., only a few ground truth values ​​are available in the tree / graph structure).

[0065] Figure 3 A framework for training a graph neural network according to an embodiment of the present invention and using it to predict disease quantification parameters of anatomical structures is shown, including two stages: offline training 310 and online prediction 320. For example... Figure 3 As shown, during offline training 310, centerline structure extraction 312 is performed on the training samples in the training image set 311. A graph representation algorithm can be used to create a graph structure representation for each training sample, and then image patch / mask patch extraction 313 is performed on the centerline structure extracted in centerline structure extraction 312. The deep neural network 316 is trained using ground truth 314, the established deep neural network 315, and the image patches / mask patches extracted in image patch / mask patch extraction 313 to obtain a trained model 317. During offline training, a database of training data annotated with ground truth can be assembled, and the constructed deep neural network can learn to transmit information (message passing) between nodes with and without ground truth through training.

[0066] In some embodiments, training can be performed using gradient-based methods (such as stochastic gradient descent (SGD)) to optimize the objective function with all parameters on the training dataset.

[0067] For example, in regression prediction problems involving parameters such as FFR, the parameters of the regression graph neural network can be optimized by minimizing any regression loss function of the regression task. For instance, this regression loss function could include the ground truth output y and the predicted value. The mean squared error (L2 loss) between them. In particular, for the training set D, the parameters (θ) of the neural network are optimized to minimize the objective function J, as shown in Equation (1) below:

[0068]

[0069] For example, in the problem of classifying and predicting patch stability levels, the parameters of the classification graph neural network can be optimized by minimizing any classification loss function of the classification task. For instance, this classification loss function could include the ground truth output y and the predicted value. The cross-entropy loss CE between them. Specifically, for the training set D, the parameters of the classification graph neural network... The objective function H is optimized to minimize the objective function H, as shown in Equation (2) below:

[0070]

[0071] In online prediction 320, the model 317 trained in offline training 310 can be used to predict disease quantification parameters of anatomical structures, specifically, such as... Figure 3 As shown in the online prediction 320, the centerline structure of the medical image to be tested 321 is extracted 322, followed by image patch / mask patch extraction 323. Then, the extracted image patch / mask patch is input into the model 317 trained in the offline training 310 for prediction (e.g., Figure 3 The prediction shown in Figure 324 is used to obtain the prediction result.

[0072] The prediction phase shown in Figure 1 is completed in online prediction 320, so that disease quantification parameters (e.g., FFR) for anatomical structures without seen data can be calculated by using a well-learned model from the offline training 310 phase.

[0073] By scheduling the training process of time-consuming and computationally-intensive learning models offline, computationally efficient deep learning models can be trained to meet the specific needs of patients and / or physicians. When disease conditions need to be predicted, existing pre-trained learning models can be used directly for prediction, which is less time-consuming and meets clinical needs.

[0074] The present invention also provides a system for predicting disease quantitative parameters of anatomical structures. Figure 4 A schematic diagram of the framework of an apparatus (which may also be configured as a system) for predicting disease quantification parameters of anatomical structures according to an embodiment of the present invention is shown. Figure 4 As shown, system 400 includes: interface 401 configured to receive clinical data to be analyzed; and processor 402 configured to execute a method for predicting disease quantification parameters of anatomical structures according to various embodiments of the present invention.

[0075] Specifically, such as Figure 4 As shown, interface 401 can receive clinical data to be analyzed sent by external medical image acquisition device 410, and send the medical image containing anatomical structures to be analyzed to processor 402. Processor 402 uses a trained GNN to predict the disease quantification parameters of the anatomical structures. Model training device 420 is used to construct the GNN, and receives a training dataset (from medical image acquisition device 410 or other training sample databases) to train the constructed GNN. It can also send the trained GNN to system 400, so that system 400 can use the trained GNN to predict the disease quantification parameters of the anatomical structures when it receives the medical image containing anatomical structures to be analyzed sent by medical image acquisition device 410.

[0076] In some embodiments, the model training apparatus 420 may additionally include input and output interfaces for communicating with a training database, a network, and / or a user interface. The user interface can be used to select a training dataset, adjust one or more parameters of the training process, select or modify the framework of the learning model, and / or manually or semi-automatically provide predictions associated with image sequences for training.

[0077] In some embodiments, the acquired feature information, along with the prediction results, can also be stored as new training samples in the model training device 420 to continuously update the training dataset and improve the performance of the trained learning network. Furthermore, the stored parameters of previously trained prediction models can be used as initial parameters for later training phases, which can significantly accelerate the training process.

[0078] In some embodiments, such as Figure 4 As shown, the system 400 may further include a memory 403, a storage 404, and a bus 405, wherein the interface 401, the memory 403, the storage 404, and the processor 402 are connected to the bus 405 and can communicate with each other through the bus 405.

[0079] Understandably, memory 403 can store trained learning models and data, such as feature information generated during the execution of a computer program. In some embodiments, memory 404 can store computer-executable instructions, such as one or more data processing programs. The computer-executable instructions can be loaded into memory 403 for execution by processor 402. In some embodiments, feature information can be extracted from data segments stored in memory 403 at different granularities. In some embodiments, feature information can be read from memory 403 one at a time or simultaneously and stored in memory 404. Processor 402 can communicate with memory 404 and be configured to execute the computer-executable instructions stored thereon. Processor 402 can be configured to implement various steps of a method for predicting disease quantification parameters of anatomical structures according to various embodiments of the present disclosure, or various steps of training a predictive learning network such as a GNN, by executing the computer-executable instructions stored in memory 404 or memory 403.

[0080] In some embodiments, the GNN of the present invention may be stored in memory 404. Optionally, the learning network may be stored in a remote device, a separate database, a distributed device, and may be used by one or more data processing programs.

[0081] In some embodiments, the model training device 420, the medical image acquisition device 410, and the system 400 may be located in the same computer or processing device.

[0082] In some embodiments, interface 401 may include, but is not limited to, network adapters, cable connectors, serial connectors, USB connectors, parallel connectors, high-speed data transmission adapters, such as fiber optic, USB 3.0, Thunderbolt interfaces, wireless network adapters, such as WiFi adapters, telecommunications (3G, 4G / LTE, etc.) adapters, etc.

[0083] In some embodiments, interface 401 may be a network interface, through which system 400 can connect to a network, such as, but not limited to, a local area network in a hospital or the Internet. The network can connect system 400 to external devices such as medical image acquisition device 410, clinical databases, and clinical data storage devices (not shown). Medical image acquisition device 410 may also be any type of imaging modality, such as, but not limited to, computed tomography (CT), digital subtraction angiography (DSA), magnetic resonance imaging (MRI), functional MRI, dynamic contrast-enhanced MRI, diffusion MRI, spiral CT, cone-beam computed tomography (CBCT), positron emission tomography (PET), single-photon emission computed tomography (SPECT), X-ray imaging, optical tomography, fluorescence imaging, ultrasound imaging, radiotherapy field imaging, etc.

[0084] In some embodiments, system 400 may be a dedicated smart device or a general-purpose smart device. For example, system 400 may be a computer customized for clinical data acquisition and clinical data processing tasks, or a server located in the cloud. For example, system 400 may be integrated into medical image acquisition device 410.

[0085] In some embodiments, processor 402 may be a processing device that includes one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), or one or more general-purpose processing devices. More specifically, processor 402 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that runs other instruction sets, or a processor that runs a combination of instruction sets. Processor 402 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or a system-on-a-chip (SoC).

[0086] In some embodiments, processor 402 may be a dedicated processor rather than a general-purpose processor. Processor 402 may include one or more known processing devices, such as those from Intel. TM Manufactured Pentium TM Core TM Xeon TMOr microprocessors of the Itanium series, etc. The disclosed embodiments are not limited to any type of processor 402 or processor circuitry, which is otherwise configured to meet computational needs for identifying, analyzing, maintaining, generating and / or providing large amounts of clinical data or manipulating such clinical data to provide disease condition predictions, or to manipulate any other type of data consistent with the disclosed embodiments. Additionally, processor 402 may include more than one processor, for example, a multi-core design or multiple processors, each having a multi-core design.

[0087] In some embodiments, memory 404 may store one or more software applications. The software applications stored in memory 404 may include, for example, an operating system (not shown) for a general computer system and an operating system for a soft control device. Furthermore, memory 404 may store the entire software application or only a portion of the software application executable by processor 402. Additionally, memory 404 may store multiple software modules for implementing various steps of a method for predicting disease quantification parameters of anatomical structures consistent with this disclosure or for processes for training a GNN.

[0088] In some embodiments, memory 404 may also store data generated / buffered during the execution of a computer program, such as clinical data, including clinical data transmitted from one or more medical image acquisition devices 410, medical image databases, image data storage devices, etc. In some embodiments, memory 404 may be a non-transitory computer-readable medium, such as read-only memory (ROM), random access memory (RAM), phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), electrically erasable programmable read-only memory (EEPROM), other types of random access memory (RAM), flash drives or other forms of flash memory, cache, registers, static memory, optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD) or other optical storage units, magnetic tape cassettes or other magnetic storage devices, or any other non-transitory medium for storing information or instructions accessible by computer devices, etc.

[0089] In some embodiments, the model training apparatus 420 may be implemented using hardware specifically programmed by software that performs the training phase. For example, the model training apparatus 420 may include a processor and a non-transitory computer-readable medium, the processor 402 of which may perform training by executing instructions of a training process stored in the computer-readable medium.

[0090] The system 400 for predicting disease quantification parameters of anatomical structures provided by this invention extracts local features from sampling points of the centerline and extracts global features based on a set of sampling points. It uses local and global features to obtain embedding features for input into a GNN, enabling seamless integration of information from sampling points of the centerline structure throughout the entire anatomical structure. By integrating local and global considerations of the sampling points into the GNN, it is possible to accurately predict disease quantification parameters of anatomical structures using only limited labeled data.

[0091] The present invention also provides a computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, implement the method for predicting disease quantification parameters of anatomical structures according to various embodiments of the present invention.

[0092] It is understood that the computer-readable storage medium includes, but is not limited to, read-only memory (ROM), random access memory (RAM), phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), electrically erasable programmable read-only memory (EEPROM), other types of random access memory (RAM), flash drives or other forms of flash memory, cache, registers, static memory, optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD) or other optical storage units, magnetic tape cassettes or other magnetic storage devices, or any other non-transitory medium for storing information or instructions that can be accessed by computer equipment, etc.

[0093] This document describes various operations or functions that can be implemented or defined as software code or instructions. Such content can be directly executable (“object” or “executable” form) source code or differential code (“incremental” or “patch” code). The software implementation of the embodiments described herein can be provided via an article of art containing code or instructions or by operating a communication interface to send data via the communication interface. Machine or computer-readable storage media can enable a machine to perform the described functions or operations and includes any mechanism that stores information in a form accessible by a machine (e.g., computing device, electronic system, etc.), such as recordable / non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash memory devices, etc.). Communication interfaces include any mechanism that engages with any of hardwired, wireless, optical, or other media to communicate with another device, such as memory bus interfaces, processor bus interfaces, internet connections, disk controllers, etc. A communication interface can be configured to provide data signals describing the software content by providing configuration parameters and / or sending signals. A communication interface can be accessed via one or more commands or signals sent to it.

[0094] The present invention also relates to a system for performing the operations described herein. This system may be specifically constructed to perform the particular methods of this disclosure, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards or optical cards, or any type of medium suitable for storing electronic instructions, wherein each medium is coupled to a computer system bus.

[0095] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its spirit and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A computer-based method for predicting disease quantitative parameters of anatomical structures, characterized in that, The method includes: Acquire medical images containing anatomical structures; Using at least one processor, a centerline structure is extracted from the medical image; and by utilizing a graph neural network (GNN), where each node of the GNN corresponds to a sampling point on the extracted centerline structure, and each edge of the GNN corresponds to a spatial constraint relationship between two sampling points, disease quantification parameters for each sampling point on the extracted centerline structure are predicted, wherein predicting the disease quantification parameters for each sampling point further includes: Based on the image patch of the sampling points, local features are extracted using a local feature encoder; A set of image patches based on a set of sampling points is used to extract global features using a global feature encoder. The set of sampling points includes the sampling points and has spatial constraints defined by the centerline structure. Embedded features are obtained based on both the local features and the global features, and the embedded features are input into the node of the GNN corresponding to the sampling point.

2. The computer-implemented method according to claim 1, characterized in that, Each of the local feature encoder and the global feature encoder employs a two-level attention mechanism including a first-level attention block and a second-level attention block, wherein the first-level attention block is configured to extract a lesion location-aware attention map, and the second-level attention block is configured to extract a pixel-to-pixel interdependent attention map.

3. The computer-implemented method according to claim 1, characterized in that, The GNN includes higher-order GNNs.

4. The computer-implemented method according to claim 1, characterized in that, The anatomical structure includes an anatomical tree structure or an anatomical diagram structure.

5. The computer-implemented method according to claim 1, characterized in that, The local feature encoder includes: a first convolutional neural network (CNN) incorporating an attention mechanism for localization, followed by a first self-attention block incorporating an attention mechanism for inter-pixel correlation.

6. The computer-implemented method according to claim 1, characterized in that, The global feature encoder includes: A second convolutional neural network (CNN) with an attention mechanism for localization is combined with a recurrent neural network (RNN) following it, and then a second self-attention block follows the RNN, the second self-attention block being combined with an attention mechanism for inter-pixel correlation, wherein the RNN is configured to accumulate information from the root of the anatomical structure to the current sampling point.

7. The computer-implemented method according to claim 5, characterized in that, The attention mechanism used for localization includes residual attention, and the first self-attention block includes a nonlocal network.

8. The computer-implemented method according to claim 1, characterized in that, Obtaining the embedding feature based on both the local feature and the global feature further includes: concatenating the local feature with the global feature to obtain the embedding feature.

9. The computer-implemented method according to claim 1, characterized in that, The GNN includes graph convolution-based neural networks or graph gating-based neural networks.

10. The computer-implemented method according to claim 1, characterized in that, The anatomical structures include vascular structures or airway structures.

11. A system for predicting disease quantitative parameters of anatomical structures, characterized in that, The system includes: The interface is configured to: acquire medical images containing anatomical structures; and A processor configured to perform a computer-implemented method for predicting disease quantification parameters of anatomical structures according to any one of claims 1-10.

12. A computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, implement a computer-implemented method for predicting disease quantification parameters of anatomical structures according to any one of claims 1-10.

Images