Method and system for anatomical landmark generation

Abstract

The present disclosure relates to a computer-implemented method and system for generating anatomical labels of an anatomical structure, the method comprising receiving an anatomical structure with extracted centerlines, or a medical image containing the anatomical structure with extracted centerlines; and predicting, by at least one processor, anatomical labels of the anatomical structure based on the centerlines of the anatomical structure using a trained deep learning network, the deep learning network being composed of a branching network, a graph neural network, a recurrent neural network and a probabilistic graphical model connected in sequence, wherein the branching network comprises at least two branch networks arranged in parallel. The computer-implemented method in the present disclosure can utilize the trained deep learning network to automatically generate anatomical labels of the whole anatomical structure in the medical image in an end-to-end manner, and the method still has high prediction accuracy and reliability in the case of large individual variation of the anatomical structure, which is conducive to helping doctors accurately and reliably diagnose.

Description

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to U.S. Provisional Application No. 63 / 178,894, filed April 23, 2021, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to the technical field of processing and analyzing medical data and medical images, and more specifically, to a computer-based method and system for generating anatomical markers for anatomical structures. Background Technology

[0004] Automatically identifying anatomical structures and assigning them correct anatomical landmarks can improve diagnostic accuracy. However, the morphology and topology of these structures can vary considerably from person to person. Therefore, the challenge of automating anatomical landmark marking stems from the significant individual variability of structures such as the coronary arteries, particularly minor branches originating from major branches.

[0005] Some existing technologies employ learning-based methods for label estimation. However, because the learning models used are either not end-to-end or require predefined features, these methods typically exhibit low reliability when applied to scenarios with significant individual variability. Furthermore, previous methods did not model the correlation between individual labels. These two points significantly limit the accuracy, robustness, and other performance characteristics of these labeling algorithms when handling scenarios with substantial individual variability. Summary of the Invention

[0006] This disclosure is provided to address the aforementioned problems existing in the prior art. This disclosure aims to provide a computer-based method and system for generating anatomical markers for anatomical structures. It can automatically generate anatomical markers for the entire anatomical structure in medical images end-to-end using a trained deep learning network. Furthermore, the computer-based method of this disclosure is robust and maintains high prediction accuracy and reliability even when anatomical structures vary significantly from individual to individual.

[0007] According to a first aspect of this disclosure, a computer implementation method for generating anatomical markers of an anatomical structure is provided, comprising receiving an anatomical structure with its centerline extracted, or a medical image containing an anatomical structure with its centerline extracted; and having at least one processor predict anatomical markers of the anatomical structure based on the centerline of the anatomical structure using a trained deep learning network, wherein the deep learning network is constructed by sequentially connecting a branching network, a graph neural network, a recurrent neural network, and a probabilistic graphical model, wherein the branching network includes at least two branching networks connected in parallel.

[0008] According to a second aspect of this disclosure, a system for generating anatomical markers for anatomical structures is provided. The system includes an interface configured to receive an anatomical structure with its centerline extracted, or a medical image containing an anatomical structure with its centerline extracted. The system also includes at least one processor configured to perform steps of a computer-implemented method for generating anatomical markers for anatomical structures according to various embodiments of this disclosure.

[0009] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein when the computer-executable instructions are executed by a processor, the steps of a computer-implemented method for generating anatomical markers of anatomical structures according to an embodiment of this disclosure are performed.

[0010] The computer implementation method for generating anatomical markers of anatomical structures according to various embodiments of this disclosure constructs a centerline map based on sampling the easily obtainable centerline of the anatomical structure. It utilizes multiple parallel branch networks to fully consider various features such as geometric and image features of the centerline map, and achieves joint embedding of multiple features based on a graph neural network. This enables automated, more accurate end-to-end prediction from anatomical structure to anatomical markers. By further utilizing a probabilistic graphical model to model the relationships between anatomical markers, a more reasonable and robust division of anatomical markers for the entire anatomical structure can be provided from a global optimization perspective. The method in this disclosure is more suitable for the accurate prediction of anatomical markers for anatomical structures with high individual variability, thereby helping to improve the diagnostic accuracy and efficiency of physicians.

[0011] The above general description and the following detailed description are exemplary and illustrative only and are not intended to limit the claimed invention. Attached Figure Description

[0012] In drawings that are not necessarily drawn to scale, the same reference numerals may describe similar parts in different views. Similar reference numerals with different letter suffixes may indicate different examples of similar components. The drawings generally illustrate various embodiments by way of example rather than limitation, and are used together with the specification and claims to illustrate the disclosed embodiments. Such embodiments are illustrative and exemplary, and are not intended to be exhaustive or exclusive embodiments of the method, apparatus, system, or non-transitory computer-readable medium having instructions for implementing the method.

[0013] Figure 1 A schematic diagram showing anatomical landmarks of an exemplary coronary artery anatomy according to an embodiment of the present disclosure.

[0014] Figure 2A flowchart illustrating a method for generating anatomical markers for anatomical structures according to an embodiment of the present disclosure is shown.

[0015] Figure 3 A schematic diagram illustrating the process of predicting anatomical landmarks using a trained deep learning network according to an embodiment of the present disclosure.

[0016] Figure 4 This diagram illustrates another process of predicting anatomical landmarks using a trained deep learning network according to an embodiment of the present disclosure.

[0017] Figure 5 A schematic block diagram of a system for generating anatomical markers for anatomical structures according to an embodiment of the present disclosure is shown.

[0018] Figure 6 A schematic diagram illustrating the workflow of a system for generating anatomical markers for anatomical structures according to an embodiment of the present disclosure is shown. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but this is not intended to limit the present invention.

[0020] Figure 1 This diagram illustrates anatomical landmarks of exemplary coronary artery structures according to embodiments of the present disclosure. It should be noted that the technical solutions of this disclosure are illustrated below using examples of coronary arteries or anatomical structures included in received medical images, but the disclosure is not limited thereto.

[0021] The coronary arteries are the arteries that supply blood to the heart. They originate from the aortic sinus at the root of the aorta, branch into left and right branches, and run on the surface of the heart. According to the classification principles of Schlesinger et al., the distribution of coronary arteries can be divided into right-dominant, balanced, and left-dominant types. Coronary arteries of different dominance types may contain different anatomical structures, and there are also various rules for anatomical marking of these structures. For example, according to the American Heart Association (AHA) classification, the coronary arteries can be roughly divided into 15 segments, that is, 15 types of anatomical markers. Figure 1 It is a specific method of anatomically marking the coronary arteries based on the AHA classification method. The rules for segmenting and marking the anatomical structures of the coronary arteries are shown in Table 1.

[0022] Table 1 Figure 1 Anatomical landmarks and marking rules of the middle coronary artery

[0023]

[0024]

[0025] Figure 1 The anatomical structure of the coronary artery shown may vary greatly among different individuals. For example, the 14th segment in the figure, that is, the second obtuse marginal branch OM2, may also be missing in some individuals, and so on. In addition, the mutual relationship between anatomical markers can also be included in the marking rules of anatomical markers. Taking Table 1 as an example, the LAD branch is connected by 5 (left main trunk LM) - 6 (proximal left anterior descending artery pLAD) - 7 (mid-segment left anterior descending artery mLAD) - 8 (distal left anterior descending artery), and the D branch includes 9 (first diagonal branch D1) and 10 (second diagonal branch D2). Among them, the D branch is necessarily located on the LAD branch. More specifically, 9 (first diagonal branch D1) originates from 6 (proximal left anterior descending artery pLAD) or 7 (mid-segment left anterior descending artery mLAD), and 10 (second diagonal branch D2) originates from 8 (distal left anterior descending artery) or the transition segment between 7 (mid-segment left anterior descending artery mLAD) and 8 (distal left anterior descending artery). In some other embodiments, the D branch may also include 9a (first diagonal branch D1a, not shown) and 10a (second diagonal branch D2a, not shown), etc. In this case, 9a (first diagonal branch D1a) should be the diagonal branch that originates from 6 (proximal left anterior descending artery pLAD) or 7 (mid-segment left anterior descending artery mLAD) and is before 8 (distal left anterior descending artery), and 10a is the second diagonal branch that originates from 8 (distal left anterior descending artery) in addition to 10 (second diagonal branch D2). In some embodiments, according to the method of the present disclosure, when determining the anatomical markers of each part of the anatomical structure, the above-mentioned two-way constraint relationship between anatomical markers can be comprehensively considered, so that from the perspective of global optimization, the prediction of the anatomical markers of the entire anatomical structure is more accurate and reliable.

[0026] It should be noted that the anatomical structure according to the embodiments of the present disclosure is not necessarily the coronary artery, but can also be any blood vessel, respiratory tract, mammary duct, etc., especially the anatomical structure with a multi-branched tree shape, which will not be listed one by one here.

[0027] Figure 2 A flowchart showing a method for generating anatomical markers of an anatomical structure according to an embodiment of the present disclosure.

[0028] First, in step 101, an anatomical structure with an extracted centerline or a medical image containing an anatomical structure with an extracted centerline can be received. In some embodiments, an anatomical structure without an extracted centerline or a medical image containing an anatomical structure without an extracted centerline can also be received, and any centerline extraction algorithm can be used to extract the centerline of the anatomical structure. At the same time, some useful data information, such as a segmentation mask, etc., can also be extracted together. The above process of centerline extraction can be automatic, semi-automatic or manual, and any applicable algorithm or method can be adopted, and the present disclosure does not limit this.

[0029] Next, in step 102, the anatomical landmarks of the anatomical structure can be predicted using a trained deep learning network based on the centerline of the anatomical structure received or extracted in step 101. In some embodiments, the deep learning network can be constructed by sequentially connecting a branching network, a graph neural network, a recurrent neural network, and a probabilistic graphical model, and the branching network can include at least two branching networks connected in parallel.

[0030] In some embodiments, when predicting anatomical landmarks based on the centerline of an anatomical structure, it is first necessary to construct a graphical representation of the centerline of the anatomical structure. Specifically, the centerline can be sampled to form a centerline graph of the anatomical structure. For example, each sampling point can be used as a node of the centerline graph (hereinafter referred to as G) (hereinafter referred to as V), and each line segment on the centerline connecting each pair of adjacent nodes in V can be used as an edge of the centerline graph (hereinafter referred to as E).

[0031] In some embodiments, for each node v in V i It can extract associated physical coordinates, image patches, and any other useful information as node v. i The expression .

[0032] In some embodiments, for example, the edge e in E can be... i This can be represented by one undirected edge or two directed edges. Directed edges can carry more information, especially in representing e. i A bidirectional constraint relationship between two connected nodes. For example, in a coronary artery tree structure, information can propagate from the root to the end of the tree, and also from the end back to the root.

[0033] Based on the aforementioned set of nodes V and edge set E, the centerline graph G can be represented as: G = (V, E), where nodes v in V are... i ∈V corresponds to the feature vector or embedding of the point on the center line, e j ∈E corresponds to directed or undirected edges between nodes, i∈[1,…,N], j∈[1,…,N-1] where N is the total number of nodes after sampling the center line.

[0034] Once the centerline map of the anatomical structure has been modeled, the anatomical landmarks of the structure can be predicted using a trained deep learning network based on this centerline map G. The specific process will be combined with... Figure 3 Detailed description.

[0035] The method for determining anatomical markers in this embodiment takes anatomical structures and medical images containing anatomical structures as input. It achieves end-to-end prediction of anatomical markers by using multiple deep learning networks, such as deep neural networks, that combine parallel and serial connections. It can learn anatomical features essential for identifying anatomical structures such as arteries without any manually defined standards or features. Compared with existing technologies, it no longer requires predefined discrete feature extraction modules. Instead, it learns and uses the aforementioned deep learning network as a whole from the perspective of global joint optimization. It can directly output the anatomical markers of various parts of the anatomical structure at the output end of the deep learning network. Furthermore, the performance of the aforementioned deep learning network in terms of accuracy and robustness will also improve as the amount of training data increases.

[0036] Figure 3 This diagram illustrates the process of predicting anatomical landmarks using a trained deep learning network according to an embodiment of the present disclosure. Figure 3 As shown, based on the centerline map G of the anatomical structure, the anatomical labels of the anatomical structure can be automatically generated end-to-end using a trained deep learning network 300. An exemplary process is as follows.

[0037] In some embodiments, the first branch network 301 first extracts the coordinate information of each node vi based on the centerline map G in step S3011, and uses the extracted coordinate information as input for embedding geometric features. Since the coordinate information can be used as a point cloud, in step S3012, any point cloud network such as PointNet or PointNet++ can be used to encode the coordinate information and generate the geometric feature embedding of each node. This disclosure does not impose specific restrictions on the point cloud network used by the first branch network 301, as long as it can generate the geometric feature embedding of each node based on the coordinate information of each node in the centerline map G.

[0038] In some embodiments, one or more other branch networks may be connected in parallel with the first branch network 301, for example... Figure 3The second branch network 302 is shown in the figure. In some embodiments, the second branch network 302 can synchronously with the first branch network 301 in step S3021, based on the centerline map G, extract the 2D / 3D image blocks or mask blocks corresponding to each node vi, and use the extracted image blocks / mask blocks as input for embedding image features. In some embodiments, for example, the optimized window width can be selected according to different anatomical locations to complete the extraction of image blocks / mask blocks, etc., and this disclosure does not impose specific limitations on this. In some embodiments, the second branch network 302 can, for example, in step S3022, use any deep learning network based on CNN (Convolutional Neural Network), GCN (Graphic Convolutional Neural Network), RNN (Recurrent Neural Network), or MLP (Multilayer Perceptron), such as ResNet, VGG, etc., to encode the image / mask information of each node and generate the image / mask feature embedding of each node.

[0039] As an example only, when the second branch network 302 selects a GCN as the deep learning network for embedding image features, the GCN can generalize the CNN architecture to non-Euclidean domains such as graphs. Graph convolution is defined directly on the graph, operating over spatial neighborhoods, and can be formally represented as Z = GCN(X, A), where X ∈ RN × C is the input, N is the node index, and C is the dimension of the feature embedding, A is the adjacency matrix used to indicate whether there are edges between nodes. In embodiments according to this disclosure, A can be determined by the centerline graph G, and Z is the output of the GCN. It should be noted that other commonly used methods in CNNs can also be used in GCNs, such as skip connections or attention mechanisms. Furthermore, in some embodiments, the second branch network 302 can also select other advanced GNN variants, such as gated GNN methods. In other embodiments, gating mechanisms such as GRU or LSTM can also be used in the propagation step to improve the long-term propagation of information on the graph structure. For example, if the edges of the graph are directed, by using a gating mechanism, the parent node can selectively combine information from each child node. More specifically, each graph unit (which can be a GRU or LSTM unit) includes input and output gates, storage units, and hidden states. Each graph unit also includes a forget gate for each child node, instead of a single forget gate. The graph unit can be any RNN unit, such as LSTM, GRU, CLSTM, CGRU, etc.

[0040] As described above, this disclosure does not impose specific restrictions on the deep learning network and feature embedding method used in the second branch network 302, as long as image information encoding can be achieved and image / mask feature embeddings of each node can be generated based on the image blocks / mask blocks corresponding to each node in the centerline graph G.

[0041] After obtaining the geometric feature embeddings and image feature embeddings of each node in the centerline graph G using the first branch network 301 and the second branch network 302 respectively, the graph neural network 303 can then be used in step S3031 to integrate the geometric and image feature embeddings of each node to obtain the joint feature embedding of each node. The method for integrating two or more feature embeddings is not specifically limited here; it could be a simple concatenation of the feature embeddings from each branch, or a weighted fusion of the feature embeddings from multiple branches based on, for example, predetermined weights, to generate the joint feature embedding. These methods are not listed here.

[0042] Next, the joint features of each node output by the graph neural network 303 can be embedded as input to the recurrent neural network 304, which then generates the anatomical labels corresponding to each node in the centerline map G of the anatomical structure. The recurrent neural network 304 can be any one of LSTM, GRU, CLSTM, CGRU, or a variant thereof.

[0043] Furthermore, probabilistic graphical models 305, such as CRF, can be used to generate the overall unit division of the anatomical structure and the corresponding anatomical labels of each unit based on the anatomical labels corresponding to each node in the centerline graph G, and according to the anatomical labels corresponding to each node, for example, by induction or clustering. The above-mentioned probabilistic graphical model can also be other models besides CRF, such as MRF (Markov Random Field) or higher-order CRF, or other variations based on it, and this disclosure does not impose any restrictions.

[0044] According to such Figure 3 The method for determining anatomical markers in the embodiments of this disclosure can not only independently consider geometric or image features in the midline map constructed based on anatomical structures, but also use GNN networks to embed multiple features into a joint feature vector. Whether it is splicing integration or considering the joint feature embedding after weighted fusion, it can provide an improvement in prediction accuracy for deep learning networks.

[0045] In some embodiments, the centerline may first be divided into multiple units. After generating joint feature embeddings for each node using the graph neural network, unit-level average pooling is performed on the joint feature embeddings of each node based on each unit of the divided centerline to generate unit-level features. Then, the recurrent neural network is used to generate unit-level anatomical labels for the centerline map based on the unit-level features. Finally, the probabilistic graphical model is used to generate anatomical labels for the anatomical structures based on the unit-level anatomical labels.

[0046] Figure 4 This diagram illustrates another process of predicting anatomical landmarks using a trained deep learning network according to an embodiment of the present disclosure. Figure 4 and Figure 3 Similarly, deep learning networks 300 all include components such as a first branch network 301, a second branch network 302, a graph neural network 303, a recurrent neural network 304, and a probabilistic graphical model 305, and the operations and steps in each component are basically the same. Figure 4 and Figure 3 The difference is that after the graph neural network 303 integrates the geometric features and image features of each node and outputs the joint feature embedding of each node in step S3031, it is not directly fed into the recurrent neural network 304. Instead, it uses the pooling unit 303' to perform unit-level average pooling on the joint feature embedding of each node in step S3032.

[0047] In some embodiments, when sampling the centerline of an anatomical structure, to obtain more accurate anatomical landmark prediction results, the sampling is usually denser, with a larger number of nodes and higher correlation between neighboring nodes. However, in anatomical structures such as blood vessels, blood vessels between two bifurcations typically belong to the same vascular branch, i.e., they have the same anatomical landmarks. Therefore, when the anatomical structure is a blood vessel, each unit of the centerline can be set to correspond to a vascular branch between two bifurcations. The division of centerline units can be done manually, automatically, or semi-automatically, without limitation.

[0048] by Figure 1 Taking the coronary artery as an example, the vascular branches between bifurcations can be used as basic units along the centerline of the coronary artery, and unit-level average pooling can be performed on the centerline map with joint embedding features. After unit-level average pooling, the unit-level features output by pooling unit 303' are input into the recurrent neural network 304. At this time, with... Figure 3 Unlike other systems, the recurrent neural network 304 will output the anatomical label sequence of each unit based on unit-level features.

[0049] As mentioned above Figure 1As shown in Table 1, branches D1 and D2, etc., lie on the LAD branch consisting of 5-6-7-8. This means that for structured labeling tasks such as anatomical label prediction, in addition to predicting its own label, it is advantageous to consider the correlation between labels in the neighborhood and the overall structure, and to jointly decode a given anatomical structure to obtain the optimal labeling structure. Therefore, in some embodiments, probabilistic graphical models (PGMs) such as Conditional Random Fields (CRFs) can be used to jointly model the correlation of labels, rather than decoding the anatomical labels of individual units independently. This allows for a more reasonable and accurate anatomical labeling from the perspective of the overall anatomical structure.

[0050] In some cases, anatomical structures can vary significantly from individual to individual. The method described in this disclosure, which models the relationships between anatomical landmarks, can largely address these individual variations. By utilizing the constraints of these relationships, the method of this disclosure enables accurate and robust predictions of anatomical landmarks throughout the entire anatomical structure, such as the entire vascular tree. This helps physicians respond accurately and efficiently to individual differences, making accurate and reliable diagnoses for different patients.

[0051] According to embodiments of this disclosure, an apparatus for generating anatomical markers of anatomical structures using a computer is also provided. The apparatus includes a memory, at least one processor, and computer-executable instructions stored in the memory and running on the at least one processor, wherein the at least one processor executes the steps of the methods for generating anatomical markers of anatomical structures described in the foregoing embodiments. In some embodiments, the above-described apparatus for generating anatomical markers of anatomical structures using a computer can be used independently or in combination with other components as a system for generating anatomical markers of anatomical structures according to embodiments of this disclosure.

[0052] According to embodiments of this disclosure, a system for generating anatomical markers of anatomical structures is also provided. The system includes an interface, a model training device, and an image processing device, wherein the interface is used to receive anatomical structures with extracted centerlines required during the training phase, or medical images containing anatomical structures with extracted centerlines, and / or to receive anatomical structures with extracted centerlines to be predicted as anatomical markers during the prediction phase, or medical images containing anatomical structures with extracted centerlines.

[0053] The system for generating anatomical markers of anatomical structures according to embodiments of the present disclosure further includes a model training device for training the deep learning network in the computer implementation method for generating anatomical markers of anatomical structures described in the foregoing embodiments during the training phase.

[0054] The system for generating anatomical markers of anatomical structures according to embodiments of the present disclosure further includes an image processing device for performing the steps of the computer implementation method for generating anatomical markers of anatomical structures described in the foregoing embodiments during the prediction phase.

[0055] Figure 5 A schematic block diagram of a system for generating anatomical markers for anatomical structures according to an embodiment of the present disclosure is shown. System 500 may include a model training apparatus 502 configured to train a deep learning network according to an embodiment of the present disclosure during a training phase, and an image processing apparatus 503 configured to perform an anatomical marker prediction task during a prediction phase. In some embodiments, the model training apparatus 502 and the image processing apparatus 503 may be located within the same computer or processing device.

[0056] In some embodiments, the image processing apparatus 503 may be a dedicated computer or a general-purpose computer. For example, the image processing apparatus 503 may be a custom computer used in a hospital to perform image acquisition or image processing tasks, or a server deployed in the cloud. The image processing apparatus 503 may include an interface 501, storage 504, memory 506, processor 508, and bus 510. The interface 501, storage 504, memory 506, and processor 508 are connected to and communicate with each other via the bus 510.

[0057] Interface 501 may include a network cable connector, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transmission adapter such as fiber optic, USB 3.0, or Xunlei, a wireless network adapter such as a WiFi adapter, or a telecommunications (3G, 4G / LTE, etc.) adapter. In some embodiments, interface 501 receives medical images containing anatomical structures from image acquisition device 505. In some embodiments, interface 501 also receives a trained deep learning network model from model training device 502.

[0058] The image acquisition device 505 is capable of acquiring images of any imaging modality, including functional MRI (e.g., fMRI, DCE-MRI, and diffusion MRI), cone-beam CT (CBCT), spiral CT, positron emission tomography (PET), single-photon emission computed tomography (SPECT), X-ray, optical tomography, fluorescence imaging, ultrasound imaging, and radiotherapy portal imaging, or combinations thereof. The method of this disclosure can be performed by a system that uses the acquired images to predict anatomical landmarks.

[0059] Storage 504 / memory 506 may be a non-transitory computer-readable medium on which computer-executable instructions may be stored, wherein when the computer-executable instructions are executed by a processor, a computer-implemented method for generating anatomical markers of anatomical structures according to embodiments of the present disclosure may be executed. Storage 504 / memory 506 may be such as read-only memory (ROM), random access memory (RAM), phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), electrically erasable programmable read-only memory (EEPROM), other types of random access memory (RAMs), flash drives or other forms of flash memory, cache, registers, static memory, read-only optical disc memory (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape or other magnetic storage devices, or any other non-transitory medium that can be used to store information or instructions accessible by a computer device.

[0060] In some embodiments, memory 504 may store trained deep learning models and data, such as centerline plots generated when a computer program is executed. In some embodiments, memory 506 may store computer-executable instructions, such as one or more image processing programs.

[0061] Processor 508 may be a processing device that includes one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc. More specifically, the processor may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a processor that runs other instruction sets, or a processor that runs a combination of instruction sets. The processor may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), system-on-a-chip (SoCs), etc. Processor 508 may be communicatively coupled to memory 506 and configured to execute computer-executable instructions stored thereon.

[0062] The model training device 502 can be implemented using hardware specifically programmed by software to perform training processing. For example, the model training device 502 may include the same processor and non-transitory computer-readable medium as the image processing device 503. The processor can perform training by executing instructions for training processing stored in the computer-readable medium. The model training device 502 may also include input and output interfaces for communicating with a training database, a network, and / or a user interface. The user interface can be used to select a training dataset, adjust parameters of one or more training processes, select or modify the framework of the learned model, and / or manually or semi-automatically provide predictions related to anatomical structures in the training images.

[0063] Figure 6 A schematic diagram illustrating the workflow of a system for generating anatomical markers for anatomical structures according to an embodiment of the present disclosure is shown.

[0064] like Figure 6 As shown, the workflow of the system for generating anatomical markers for anatomical structures according to embodiments of this disclosure can be divided into a training phase and a prediction phase, and the specific workflow is as follows:

[0065] In some embodiments, training phase 61 is an offline process. In this phase, training images can first be received in step S611. These images could be anatomical structures with extracted centerlines, or medical images containing anatomical structures with or without extracted centerlines. If the centerline of the anatomical structure is not extracted, the centerline can be extracted from the training image in step S612 using any applicable centerline extraction algorithm. Next, in step S613, the deep learning network to be trained can be modeled. The deep learning network consists of a branching network, a graph neural network, a recurrent neural network, and a probabilistic graphical model sequentially connected in series. The branching network includes at least two branching networks connected in parallel. In step S613, a graph representation algorithm is also used to automatically extract image blocks / mask blocks, coordinates, and other data information corresponding to each sampling point from the centerline, creating a centerline map of the anatomical structure. Feature embedding is then performed on each node in the centerline map. In training phase 61, ground truth values ​​of anatomical landmarks for the sample anatomy can be received in step S614, or, in this phase, the system assembles a database of training data for the sample anatomy labeled with ground truth values. Next, in step S615, the modeled deep learning network can be trained based on the centerline map after feature embedding and the ground truth values ​​of the anatomical landmarks, and the trained deep learning network can be obtained in step S616. When training the end-to-end deep learning network model, gradient-based methods (e.g., SGD, Adam, etc.) can be used to optimize the objective function J relative to the model parameters of all networks and models on the training dataset. The parameters (θ) of the deep learning network model can be minimized by minimizing the ground truth values ​​y and predicted outputs of each node on the centerline map. The mean squared error between the training set D can be used for optimization. In particular, for the training set D, the parameters (θ) can be optimized to minimize the objective function J, where J can be any classification loss or AUC loss.

[0066] The prediction phase 62 can be processed online. In some embodiments, a new test image of the anatomical landmarks to be predicted can first be received in step S621. The test image should contain anatomical structures with or without extracted centerlines. When the anatomical structures in the received test image do not have extracted centerlines, the centerlines of the anatomical structures in the test image can be extracted in step S622. Then, in step S623, the anatomical landmarks of the entire anatomical structure in the new test image can be calculated using the deep learning network trained in the training phase 61.

[0067] Various modifications and alterations can be made to the methods, apparatus, and systems disclosed herein. In view of the description and practice of the disclosed systems and related methods, other embodiments can be derived by those skilled in the art. Each claim of this disclosure is to be understood as an independent embodiment, and any combination thereof is also used as an embodiment of this disclosure, and such embodiments are considered to be included in this disclosure.

[0068] The descriptions and examples are to be considered exemplary only, and the true scope is indicated by the appended claims and their equivalents.

Claims

1. A computer-based method for generating anatomical markers for anatomical structures, characterized in that, include: Receive anatomical structures with their center lines extracted, or medical images containing anatomical structures with their center lines extracted; and consisting of at least one processor, The centerline is sampled to form a centerline diagram of the anatomical structure, wherein each sampling point is used as a node of the centerline diagram of the anatomical structure, and each line segment on the centerline connecting each pair of adjacent nodes is used as an edge of the centerline diagram. Based on the centerline map, a trained deep learning network is used to predict the anatomical markers corresponding to each node in the centerline map. Based on these anatomical markers, the overall anatomical structure is divided into units, and the anatomical markers corresponding to each unit are generated. The deep learning network is constructed by sequentially connecting branching networks, graph neural networks, recurrent neural networks, and probabilistic graphical models. The branching network includes a first branch network and a second branch network connected in parallel. The first branch network is used to generate geometric feature embeddings of each node based on the coordinate information of each node, and the second branch network is used to generate image feature embeddings of each node based on the image blocks or mask blocks corresponding to each node. The graph neural network is used to generate joint feature embeddings for each node based on the geometric feature embeddings and the image feature embeddings; The recurrent neural network is used to generate anatomical markers corresponding to each node in the centerline diagram based on the joint feature embedding. The probabilistic graphical model is used to generate anatomical markers for the anatomical structure based on the anatomical markers corresponding to each node in the centerline diagram, and to generate the overall unit division of the anatomical structure and the anatomical markers corresponding to each unit.

2. The computer implementation method according to claim 1, characterized in that, The first branch network is a point cloud neural network.

3. The computer implementation method according to claim 1, characterized in that, The second branch network is one of CNN, RNN or MLP.

4. The computer implementation method according to claim 1, characterized in that, The recurrent neural network is one of LSTM, GRU, CLSTM or CGRU.

5. The computer implementation method according to claim 1, characterized in that, The anatomical structures include at least one of the following: blood vessels, respiratory tract, or mammary ducts.

6. The computer implementation method according to claim 1, characterized in that, The edges of the centerline diagram are directed edges.

7. The computer implementation method according to claim 1, characterized in that, The computer implementation method further includes: Receive training data consisting of sample anatomical structures with extracted sample centerlines, or sample medical images containing sample anatomical structures with extracted sample centerlines, and ground truth values ​​of anatomical markers of the sample anatomical structures. The deep learning network is trained using the training data by jointly optimizing the parameters of at least two branch networks, graph neural networks, recurrent neural networks, and probabilistic graphical models.

8. A system for generating anatomical markers for anatomical structures, the system comprising: An interface configured to receive anatomical structures with their centerlines extracted, or medical images containing anatomical structures with their centerlines extracted. At least one processor is configured to perform a computer-implemented method for generating anatomical markers of anatomical structures according to any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, a computer-implemented method for generating anatomical markers for anatomical structures according to any one of claims 1 to 7 is performed.

Images