Blood vessel segmentation method based on medical image

Abstract

The present application relates to the field of medical images, and discloses a blood vessel segmentation method based on a medical image, comprising: using a segmentation model to perform preliminary blood vessel segmentation on a three-dimensional medical image to generate a preliminary blood vessel segmentation result; localizing a blood vessel on the basis of the preliminary blood vessel segmentation result, and generating a first graph structure; encoding the first graph structure by using a dual-stream encoder based on a graph channel attention network and a graph spatial attention network, and outputting a result to a dual-stream decoder based on the graph channel attention network and the graph spatial attention network by means of a graph convolutional network for decoding, so as to obtain a second graph structure, wherein the graph channel attention network focuses on the mutual influence between node feature channels in the graph structure, and the graph spatial attention network focuses on node information and a relationship thereof in a spatial channel of the graph structure; and generating an updated blood vessel segmentation result on the basis of the second graph structure. The present application can more accurately detect highly complex blood vessels, enhancing the feature representation capability of networks.

Description

A blood vessel segmentation method based on medical images TECHNICAL FIELD

[0001] The present application relates to the technical field of medical images, more particularly to a blood vessel segmentation method based on medical images. BACKGROUND

[0002] This section is intended to provide background information to facilitate a better understanding of embodiments of the present application recited in the claims. The content of this section is not to be taken as an acknowledgement that it is prior art merely by virtue of its inclusion in this section.

[0003] In traditional clinical diagnosis, doctors rely on their experience to find lesions or observe tissues and organs according to medical images of patients to make surgical plans. With the improvement of the accuracy of computer tomography (CT) and magnetic resonance (MR) technology, the number of layers of medical images is also increasing, which greatly increases the workload of doctors reading films. At the same time, the analysis and diagnosis of medical images are highly dependent on the subjective judgment of doctors, which increases the possibility of misdiagnosis or missed diagnosis. For target regions, especially for irregularly distributed and complex topological structure blood vessels, manual annotation of liver blood vessels layer by layer is very time-consuming and prone to errors. Therefore, the proposal of automatic segmentation method of blood vessels or other target regions can not only reduce the workload of doctors, but also assist doctors in making surgical plans by precise quantification of target regions.

[0004] Various structures in human physiology follow a tubular morphology, such as intrathoracic airways, retinal blood vessels, cardiovascular and hepatic vessels, etc. At the same time, different vascular structures are also presented by different modalities of medical images, such as CT, MR, fundus images, OCT, etc. Fast and accurate segmentation and accurate quantitative analysis of blood vessels or tubular structures in medical images can assist doctors in understanding disease diagnosis and making surgical plans. For example, airway structural changes are usually associated with a series of chronic lung diseases, including asthma, chronic obstructive pulmonary disease (COPD), cystic fibrosis (CF) and coronavirus disease (COVID19); the morphological structure of retinal blood vessels can be used to diagnose hypertension, diabetes and atherosclerosis; segmentation and quantification of liver blood vessels can help doctors accurately insert needles by avoiding blood vessel areas in preoperative planning of liver tumor ablation surgery. However, due to the large difference in image imaging signal intensity and the presence of much noise, and the large difference in blood vessel distribution, topological structure and thickness of different patients, and the point-like blood vessel structure on 2D slices is easily confused with noise, and the small diameter blood vessels are blurred on the image, resulting in poor connectivity of the segmentation results, which brings great trouble to traditional threshold-based, region growing, prior knowledge-based and model-based methods.

[0005] In recent years, deep learning methods have shown excellent performance in medical image segmentation. U-Net is considered as the backbone network due to its simple structure and excellent performance in medical image segmentation, and various U-shaped network variants have been derived. Benefiting from the long-range dependency extraction capability of Transformer, some medical image segmentation methods based on Transformer have surpassed traditional CNN methods. However, the receptive fields of CNN and Transformer-based segmentation methods are fixed 8-neighbor grids, which do not allow irregular data structures as input. Therefore, for the irregular structure of the blood vessel segmentation task, the connectivity problem of the blood vessel cannot be fully considered. Graph neural network (GNN) allows irregular receptive fields and is suitable for elongated shape segmentation tasks such as blood vessels. However, the current GNN-based methods only combine graph neural networks with CNNs, and the feature extraction and expression capabilities of the graph structure are insufficient.

[0006] Specifically, at present, there are the following severe challenges for the segmentation task of human tubular structure:

[0007] The contrast between the blood vessels and the surrounding tissues of the CT image without enhancement is low, and it is almost impossible to determine the position of the blood vessels. The MR image has large signal intensity difference and a lot of noise, which makes it difficult to distinguish the blood vessel boundary and forms many high-difficulty blood vessels that are difficult to detect;

[0008] The distribution, topological structure and thickness of blood vessels of different patients differ greatly, and the data situation is complex;

[0009] The region growing method and the variational level set method are both semi-automatic segmentation methods, which need to select the initial seed point manually, so the time cost is large. At the same time, the blood vessel topological structure is complex and unevenly distributed, and the manual selection of seed points on the two-dimensional image converted from the three-dimensional image makes the manual operation more difficult;

[0010] For the MR image without blood vessel enhancement, the point-like blood vessel structure on the 2D slice is easily confused with noise, and the small diameter blood vessel is blurred on the image, resulting in poor connectivity of the segmentation result; the receptive fields of the segmentation methods based on convolutional neural network (CNN) and Transformer are fixed 8-neighbor grids, which do not allow irregular data structures as input, i.e. for the irregular structure of the blood vessel segmentation task, the connectivity problem of the blood vessel cannot be fully considered;

[0011] The current graph neural network (GNN) based method only focuses on the spatial relationship between nodes, and the expression ability of the nodes is insufficient.

[0012] SUMMARY

[0013] The application aims to provide a medical image-based blood vessel segmentation method which can more accurately detect high-difficulty blood vessels.

[0014] The application discloses a medical image-based blood vessel segmentation method, comprising:

[0015] performing preliminary blood vessel segmentation on a three-dimensional medical image using a segmentation model to generate a preliminary blood vessel segmentation result;

[0016] locating blood vessels according to the preliminary blood vessel segmentation result to generate a first graph structure;

[0017] encoding the first graph structure using a double-flow encoder based on a graph channel attention network and a graph space attention network, and outputting the encoding result to a double-flow decoder based on the graph channel attention network and the graph space attention network through a graph convolution network to decode the encoding result to obtain a second graph structure; wherein the graph channel attention network focuses on the mutual influence between node feature channels in the graph structure, and the graph space attention network focuses on node information and its relationship in the spatial channel of the graph structure; and

[0018] generating an updated blood vessel segmentation result according to the second graph structure.

[0019] In a preferred example, the step of generating a first graph structure further comprises:

[0020] dividing the preliminary blood vessel segmentation result into a plurality of superpixel regions through clustering of pixels, constructing graph nodes with the centers of the superpixel regions, and constructing graph edges with the plurality of graph nodes; wherein the clustering is based on a cosine distance difference as a direction metric and a pixel gray value difference as a distance metric.

[0021] In a preferred example, the graph channel attention network further comprises:

[0022] inputting graph node features and applying a first multi-layer perceptron for calculation;

[0023] compressing the spatial dimension of the input feature channel through average pooling and maximum pooling to obtain two feature descriptors, respectively, and performing matrix multiplication operation on the two feature descriptors and the graph node features to calculate attention;

[0024] applying a second multi-layer perceptron to the two feature descriptors obtained through the attention calculation, respectively, and using element-wise addition on the output results of the two parts to obtain graph channel attention features.

[0025] In a preferred example, a U-shaped network is used in the step of obtaining a second graph structure:

[0026] The U-shaped network has an input side and an output side, the input side includes one or more double-stream encoders, the output side includes one or more double-stream decoders, the double-stream encoders and the double-stream decoders are symmetrically arranged in pairs, each pair of double-stream encoders and double-stream decoders is located in the same layer, and the bottom is the graph convolution network;

[0027] In the U-shaped network, the first graph structure is the input of the top-layer double-stream encoder, the input side is connected in the direction from the top layer to the bottom layer, the output side is connected in the direction from the bottom layer to the top layer, and the same layer is connected in the direction from the double-stream encoder to the double-stream decoder, and the output result of the top-layer double-stream decoder is the second graph structure.

[0028] In one preferred embodiment, the step of obtaining the second graph structure further comprises:

[0029] The first graph structure is input into the top-layer double-stream encoder, and the first graph spatial attention feature is generated in the graph spatial attention network and the first graph channel attention feature is generated in the graph channel attention network;

[0030] The first graph spatial attention feature and the first graph channel attention feature are input into the feature fusion network to generate a feature fusion result, and the first encoding feature is obtained by splicing;

[0031] The multi-layer perceptron is applied to the first encoding feature to obtain the second encoding feature, the first encoding feature and the second encoding feature are multiplied in matrix to calculate attention, and the top-layer input graph structure is obtained by graph pooling calculation;

[0032] After passing through N-1 double-stream decoders, the bottom-layer input graph structure is obtained, which is applied to the graph convolution network to obtain the bottom-layer output graph structure;

[0033] The bottom-layer output graph structure is input into the Nth double-stream decoder, the bottom-layer output graph structure is graph de-pooling calculated, and the second graph spatial attention feature is generated in the graph spatial attention network and the second graph channel attention feature is generated in the graph channel attention network by inputting the feature fusion result together;

[0034] The first decoding feature is obtained by splicing the second graph spatial attention feature and the second graph channel attention feature, the multi-layer perceptron is applied to the first decoding feature to obtain the second decoding feature, and the first decoding feature and the second decoding feature are multiplied in matrix to obtain the N-1 layer output graph structure;

[0035] After passing through N-1 double-stream decoders, the second graph structure is obtained, wherein N is a positive integer.

[0036] In one preferred embodiment, the step of generating the feature fusion result further comprises:

[0037] The graph spatial attention features are respectively input into a first forward mapping function, a first convolution combination, a first backward mapping function, and a first curve enhancement to obtain a first spatial feature, and the graph spatial attention features are respectively input into a second forward mapping function, a second convolution combination, a second backward mapping function, and a second curve enhancement to obtain a second spatial feature;

[0038] The graph channel attention features are applied to a multi-layer perception and a third curve enhancement to obtain a channel feature;

[0039] The channel feature is respectively spliced with the first spatial feature and the second spatial feature, and matrix multiplication is performed on the two parts of results to obtain a feature fusion result.

[0040] In one preferred embodiment, the graph pooling calculation and the graph unpooling calculation comprise:

[0041] The graph pooling calculation comprises sampling graph nodes using a graph pooling operation, projecting all node features to one dimension using a trainable projection vector, and selecting nodes with the largest scalar projection value to form a new graph while retaining the position information of the unselected nodes; and

[0042] The graph unpooling calculation comprises restoring the original graph structure using the result of the graph pooling calculation and the position information of the nodes.

[0043] The application further discloses a blood vessel segmentation device based on a medical image, comprising:

[0044] a memory for storing computer executable instructions; and

[0045] a processor coupled to the memory for implementing the steps in the method as described above when executing the computer executable instructions.

[0046] The application further discloses a computer readable storage medium having computer executable instructions stored therein, the computer executable instructions being executed by a processor to implement the steps in the method as described above.

[0047] The application further discloses a computer program product comprising computer executable instructions, the computer executable instructions being executed by a processor to implement the steps in the method as described above.

[0048] In the embodiments of the application, powerful pattern recognition is achieved through a deep learning method, and high-difficulty blood vessels such as small, tortuous or blurred blood vessels can be more accurately detected;

[0049] Further, the application has better generalization ability and can better adapt to different data situations.

[0050] Further, the application can automatically learn feature representation, without manually designing feature extractors, reducing the need for manual feature engineering.

[0051] Further, the application can effectively capture the topology and relationship between nodes in the graph data, effectively integrate local and global information, and represent the connectivity of blood vessels.

[0052] Further, the application can extract graph information from multiple scales, enhancing the feature expression ability of the network.

[0053] Each of the technical features disclosed in the above summary, each of the technical features disclosed in the following various embodiments and examples, and each of the technical features disclosed in the accompanying drawings, can be freely combined with each other to form various new technical solutions (which should all be considered to have been described in this specification), unless such a combination of technical features is technically infeasible. For example, in one example, features A+B+C are disclosed, in another example, features A+B+D+E are disclosed, features C and D are equivalent technical means that play the same role, and can only be used in one way in technology, and feature E can be combined with feature C in technology. Therefore, the scheme of A+B+C+D should not be considered to have been described because it is technically infeasible, and the scheme of A+B+C+E should be considered to have been described. BRIEF DESCRIPTION OF DRAWINGS

[0054] FIG. 1 is a schematic diagram of the overall flow according to one embodiment of the application.

[0055] FIG. 2 is a schematic diagram of obtaining superpixel segmentation according to one embodiment of the application.

[0056] FIG. 3 is a schematic diagram of a dual-stream encoder and a dual-stream decoder according to one embodiment of the application.

[0057] FIG. 4 is a schematic diagram of a graph channel attention module according to one embodiment of the application.

[0058] FIG. 5 is a schematic diagram of a feature fusion module according to one embodiment of the application.

[0059] FIG. 6 is a comparison chart of quantitative results of selecting nine segmentation models as a baseline on MR data sets according to one embodiment of the application.

[0060] FIG. 7 is a visualization chart of performance comparison of using UNETR++ as the first stage segmentation network on MR data sets according to one embodiment of the application.

[0061] FIG. 8 is a visualization of a comparison of the MR dataset and other methods according to an embodiment of the present application.

[0062] FIG. 9 is a visualization of a comparison of the CT dataset and other methods according to an embodiment of the present application.

[0063] FIG. 10 is a flowchart according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without some or all of these specific details. In other instances, well known process steps have not been described in detail in order not to unnecessarily obscure the present application.

[0065] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0066] The first embodiment of the present application relates to a medical image-based blood vessel segmentation method, the flowchart of which is shown in FIG. 10, which includes the following steps:

[0067] performing preliminary blood vessel segmentation on the three-dimensional medical image using a segmentation model to generate a preliminary blood vessel segmentation result;

[0068] locating the blood vessels according to the preliminary blood vessel segmentation result to generate a first graph structure;

[0069] encoding the first graph structure using a dual-stream encoder based on a graph channel attention network and a graph spatial attention network, and outputting the encoding result to a dual-stream decoder based on the graph channel attention network and the graph spatial attention network through a graph convolution network for decoding to obtain a second graph structure; wherein the graph channel attention network focuses on the mutual influence between the node feature channels in the graph structure, and the graph spatial attention network focuses on the node information and its relationship in the spatial channel of the graph structure; and

[0070] generating an updated blood vessel segmentation result according to the second graph structure.

[0071] In an embodiment, taking MR and CT image liver blood vessel segmentation as an example, as shown in FIG. 1, the medical image-based blood vessel segmentation method proposed in the present application is mainly divided into two stages. The first stage includes performing preliminary blood vessel segmentation on the three-dimensional medical image using a segmentation model to generate a preliminary blood vessel segmentation result; locating the blood vessels according to the preliminary blood vessel segmentation result to generate a first graph structure; the first stage will be described in combination with relevant features and more specific steps 1, 2 and 3.

[0072] The second stage includes encoding the first graph structure using a dual-stream encoder based on a graph channel attention network and a graph spatial attention network, outputting the encoding result to a dual-stream decoder based on a graph channel attention network and a graph spatial attention network through a graph convolution network to obtain a second graph structure, and generating an updated blood vessel segmentation result according to the second graph structure. The second stage will be described in combination with related features and more specific step 4.

[0073] The first stage:

[0074] In one embodiment, the step of generating the preliminary blood vessel segmentation result further includes:

[0075] Data preprocessing is performed, and a segmentation model including an encoder and a decoder with multiple skip connections is used to perform preliminary blood vessel segmentation on the three-dimensional medical image to generate a preliminary blood vessel segmentation result.

[0076] In one embodiment, the step of generating the preliminary blood vessel segmentation result specifically includes:

[0077] Step 1: Data preprocessing

[0078] Step 1.1: According to the liver mask, the liver region bounding box is extracted, and 5 pixels are reserved on both sides in each direction;

[0079] Step 1.2: In order to reduce the calculation cost of graph generation, the cross section of the image is resampled to 256x256. At the same time, the z-axis spacing of the MRI image is unified to 2mm, and the CT image is resampled to 1.6mm;

[0080] Step 1.3: Since the MR data collected by the present application comes from different centers, in order to eliminate field bias, data augmentation is performed, including rotation, scale change and elastic deformation. The same data augmentation strategy is adopted on the CT data set to maintain the same input conditions;

[0081] Step 1.4: All image pixel values are normalized to zero mean and unit standard deviation;

[0082] Step 1.5: When training and testing the network, the data is subjected to five-fold cross-validation. For MR images, the data set is divided according to the number of MR images from different centers.

[0083] Step 2: Segmentation model for blood vessel positioning

[0084] The application selects nine segmentation models widely used in the field of medical image segmentation and outstanding in effect as the baseline of this stage, including 2D U-Net, U-Net++, TransUNet, Swin-UNet and MISSFormer based on 2D, and 3D U-Net, nnFormer, UNETR and UNETR++ based on 3D. The UNETR++ model with the best segmentation result is selected as the baseline of the segmentation network in the first section through verification experiments. The UNETR++ model includes multiple encoders and decoders, each encoder includes multiple convolutional layers, pooling layers and up-sampling layers, the decoder includes multiple convolutional layers and up-sampling layers, and the encoders and decoders are connected by skip connection. The specific implementation steps of taking UNETR++ as the first stage segmentation model are as follows:

[0085] Step 2.1: The preprocessed image training set is divided into 64x64x16 3D patches and input into UNETR++ for training;

[0086] Step 2.2: UNETR++ is an encoder-decoder structure, the output of the encoder is connected to the decoder through a skip connection, and finally a 3x3x3 and a 1x1x1 convolutional block is used to produce the final segmentation mask;

[0087] Step 2.3: The loss function adopts the combination of binary cross entropy and dice loss, binary cross entropy evaluates the prediction accuracy of the model by measuring the probability distribution of the predicted value and the true label, and is more effective when dealing with class imbalance:

[0088] Dice loss is used to measure the similarity between the predicted value and the true label, which pays more attention to the accuracy of the target segmentation boundary:

[0089] The overall loss function is combined by two parts of loss:

[0090] In one embodiment, as shown in FIG. 2, the step of generating the first graph structure further includes:

[0091] The preliminary blood vessel segmentation result is divided into multiple superpixel regions by clustering of pixels, and the centers of the superpixel regions constitute graph nodes, and the graph edges are constructed by multiple graph nodes; wherein the clustering is based on the cosine distance difference as the direction measure and the pixel gray value difference as the distance measure.

[0092] When the graph nodes are constructed, as shown in FIG. 1, a first loss is formed between the graph nodes and the reference true value, and the iteration is performed until the first loss converges, and the first graph structure is obtained.

[0093] In one embodiment, the step of generating the first graph structure specifically comprises:

[0094] Step 3: Graph structure generation

[0095] Step 3.1: Graph node construction:

[0096] The GNN needs to undertake high time and space costs for voxel-level operation on the 3D image. Therefore, the present application first performs superpixel segmentation on the 3D image, and then constructs graph nodes based on the centers of the grouped superpixel regions.

[0097] Step 3.1.1: First, uniformly sample k points in three-dimensional space as initial clustering centers, and then each pixel is associated with the nearest clustering center through a specially designed metric function;

[0098] Step 3.1.2: The traditional simple linear clustering method (SLIC) uses a combination of image intensity and Euclidean distance to measure pixels and clustering centers. However, the Euclidean distance pays more attention to the absolute difference in distance, and has limitations for tubular structures with anisotropy in three-dimensional space. Therefore, the present application uses the cosine distance which pays attention to the difference in direction as the directional measurement, and combines the pixel gray value difference as the distance measurement. The specific formula is as follows: d gray (A, B) = |I A -I B | d= d gray (A, B) + d cos (A, B)

[0099] Step 3.1.2: Iteratively repeat the assignment and update the above steps to guide the error convergence;

[0100] Step 3.1.3: Calculate the mean value of the superpixel region pixels according to the initial segmentation blood vessel probability map to determine the probability of each region being a blood vessel. And set a lower threshold (0.2) to give more opportunities to the superpixel regions that are not predicted as blood vessels in the initial segmentation, as candidates for the second-stage blood vessel connectivity enhancement network.

[0101] Step 3.2: Graph edge and initial feature construction:

[0102] Step 3.2.1: In order to reduce the computational cost, the present application only considers the local connection of the nodes. Each node is only connected to the adjacent nodes with a cosine distance less than a certain threshold, i.e. dcos < Tcos, where Tcos is set to 0.4;

[0103] Step 3.2.2: The initial graph node feature is the average feature of all pixels in the superpixel region, including 32 first-stage network features, node blood vessel probability and node initial segmentation result, so the initial feature dimension is 34.

[0104] The second stage comprises:

[0105] In one embodiment, the step of obtaining the second graph structure uses a U-shaped network, as shown in FIG. 1, which is described from the perspective of structure as follows:

[0106] The two sides of the U-shaped network are an input side and an output side, the input side includes one or more double-flow encoders, and the output side includes one or more double-flow decoders, the double-flow encoders and the double-flow decoders are symmetrically and pairwisely distributed, each pair of double-flow encoders and double-flow decoders is located in the same layer, and the bottom is a graph convolutional network.

[0107] In the U-shaped network, the first graph structure is the input of the top-layer double-flow encoder, the input side is connected in the direction from the top layer to the bottom layer, the output side is connected in the direction from the bottom layer to the top layer, the same layer is connected in the direction from the double-flow encoder to the double-flow decoder, and the output result of the top-layer double-flow decoder is the second graph structure. The structure inside and the relationship between each other of the double-flow encoder and the double-flow decoder in the same layer are shown in FIG. 3.

[0108] In one embodiment, as shown in FIG. 1, the step of obtaining the second graph structure further comprises:

[0109] The first graph structure is input into the top-layer double-flow encoder, and is respectively input into the graph spatial attention network to generate a first graph spatial attention feature and into the graph channel attention network to generate a first graph channel attention feature;

[0110] The first graph spatial attention feature and the first graph channel attention feature are input into a feature fusion network to generate a feature fusion result, and are spliced to obtain a first encoding feature;

[0111] A multi-layer perceptron is applied to the first encoding feature to obtain a second encoding feature, the first encoding feature and the second encoding feature are matrix-multiplied to calculate attention, and graph pooling calculation is performed to obtain a top-layer input graph structure;

[0112] After passing through N-1 double-flow decoders, a bottom-layer input graph structure is obtained, the bottom-layer input graph structure is applied to the graph convolutional network to obtain a bottom-layer output graph structure;

[0113] The bottom-layer output graph structure is input into the Nth double-flow decoder, the bottom-layer output graph structure is subjected to graph unpooling calculation, and is input into the graph spatial attention network together with the feature fusion result to generate a second graph spatial attention feature and into the graph channel attention network to generate a second graph channel attention feature;

[0114] The second graph space attention feature and the second graph channel attention feature are spliced to obtain a first decoding feature, a multi-layer perceptron is applied to the first decoding feature to obtain a second decoding feature, and the first decoding feature and the second decoding feature are multiplied in a matrix to obtain an N-1 layer output graph structure, where N is a positive integer.

[0115] After passing through N-1 double-flow decoders, a second graph structure is obtained.

[0116] In one embodiment, the step of obtaining the second graph structure and the step of generating an updated blood vessel segmentation result according to the second graph structure specifically include:

[0117] Step 4: Blood vessel connectivity enhancement network in the second stage

[0118] The input of the second stage network is the first graph structure generated in step 3, and the network structure includes a same-layer double-flow graph attention (TGA) encoder, a TGA decoder, and a flow for transmitting graph structure information.

[0119] Step 4.1: Double-flow graph attention encoder-decoder

[0120] The graph attention network (GAT) only focuses on the attention between the spatial channel nodes of the graph, and ignores the mutual influence between the feature channels of the graph nodes. Therefore, the graph channel attention network (GCT) is proposed in the present application, which is connected in parallel with the GAT as a parallel double-flow feature extraction structure, and extracts the spatial attention and the channel attention of the graph respectively; it is embodied that the present application can extract the graph information from multiple scales, and the feature expression ability of the network is strengthened.

[0121] In one embodiment, as shown in FIG. 4, the graph channel attention network further includes:

[0122] The input graph node feature is calculated by applying a first multi-layer perceptron;

[0123] The spatial dimension of the compressed input feature channel is calculated by average pooling and maximum pooling, and two feature descriptors are obtained respectively, and the two feature descriptors are multiplied with the graph node feature to calculate the attention.

[0124] A second multi-layer perceptron is applied to the two feature descriptors calculated by the attention respectively, and the output results of the two parts are added element by element to obtain the graph channel attention feature.

[0125] In one embodiment, the graph channel attention network specifically includes:

[0126] Step 4.1.1: The graph channel attention mechanism is first proposed in the present application:

[0127] The present application generates a channel attention map by utilizing the inter-channel relationship of features. In order to effectively calculate the channel attention, the spatial dimension of the input feature channel is compressed by average pooling and maximum pooling calculation. The input graph vertex feature is first calculated by a multilayer perceptron C1 with a hidden layer:

[0128] Wherein σ represents an activation function, W1 and W2 represent weight matrices in the multilayer perceptron C1 respectively; then maximum pooling and average pooling operations are adopted in the graph channel dimension respectively, and feature descriptors Fmax and Favg are obtained:

[0129] Finally, a shared multilayer perceptron C2 with a hidden layer is applied to the two feature descriptors, and the element-wise addition of the two outputs is obtained to obtain the final channel attention feature:

[0130] Wherein, W3 and W4 represent weight matrices in the multilayer perceptron C1 respectively.

[0131] In one embodiment, the graph spatial attention network specifically comprises:

[0132] Step 4.1.2: Graph spatial attention feature:

[0133] Given the input graph structure G l (V, E) of the l-th layer, the vertex i and its neighbor vertex j and the graph node feature f i l ∈R C , C is the number of input vertex feature channels. GAT updates the node feature to obtain the graph node feature of the 1+1-th layer C is the number of vertex feature channels of the 1+1-th layer. The attention coefficient between the vertex i and the vertex j after the normalization by the softmax function is as follows:

[0134] Wherein W is a shared weight matrix, a T is a weight vector that maps high-dimensional features to a real number. || is a splicing operation. Ni is the neighbor vertex of the vertex i in the graph; then the influence of the neighbor vertex on the vertex i is transmitted by weighted summation, and the learning feature expression is stabilized by multi-head attention, which is defined as follows:

[0135] Wherein M is the number of multi-head attention, and W m respectively represent the attention coefficient and the weight matrix of the m-th attention head.

[0136] In one embodiment, the step of splicing the first encoded features further comprises a self-attention mechanism, which specifically comprises:

[0137] Step 4.1.3: Self-attention mechanism:

[0138] When splicing the spatial attention and channel attention features, the features across views can differ in numerical scale, causing the difference to be amplified in the subsequent feature updating process. Therefore, the application adopts a self-attention mechanism to enable the network to adaptively balance the contributions of GAT and GCT to updating the features, which is expressed in the formula as follows:

[0139] where represents the multiplication between elements, and β i represents the self-attention coefficient, and MLP SA represents a multi-layer perceptron for the self-attention mechanism.

[0140] In one embodiment, as shown in FIG. 5, the step of generating the feature fusion result further comprises:

[0141] The graph spatial attention features are respectively input into a first forward mapping function, a first convolution combination, a first backward mapping function, and a first curve enhancement to obtain a first spatial feature, and the graph spatial attention features are respectively input into a second forward mapping function, a second convolution combination, a second backward mapping function, and a second curve enhancement to obtain a second spatial feature.

[0142] The graph channel attention features are applied with a multi-layer perceptron and a third curve enhancement to obtain a channel feature.

[0143] The channel feature is spliced with the first spatial feature and the second spatial feature, respectively, and the two results are multiplied to obtain the feature fusion result.

[0144] In one embodiment, the step of generating the feature fusion result specifically comprises:

[0145] Step 4.2: Feature fusion:

[0146] The application proposes to effectively fuse the graph spatial attention features extracted by GAT and the graph channel attention features extracted by GCT. For the graph spatial attention features updated by GAT, the features are first converted from the node domain to the spatial domain based on CNN through a forward mapping function f(), then the spatial domain features are resampled and calculated through an extended three-dimensional convolution combination 3x1x3 and 3x3x1, respectively, and finally the spatial domain features are converted back to the node domain through a backward mapping function b():

[0147] The graph channel attention features updated by GCT are concatenated with the graph spatial features enhanced by curve after being calculated by an MLP, and the last two parts of the features are multiplied by matrix elements to obtain fused features, which are passed to the decoder as supplementary information, as follows:

[0148] wherein, represents a feature channel concatenation operation, and represents element multiplication;

[0149] In an embodiment, the graph pooling calculation and the graph unpooling calculation include:

[0150] The graph pooling calculation includes sampling graph nodes using a graph pooling operation, projecting all node features to one dimension using a trainable projection vector, and selecting nodes with the largest scalar projection value to form a new graph while retaining the position information of the unselected nodes.

[0151] The graph unpooling calculation includes restoring the original graph structure using the results of the graph pooling calculation and the position information of the nodes, specifically, using the graph structure recorded by the same layer encoder before the graph pooling calculation, restoring the pooled nodes in the graph unpooling process, and setting the restored node features to 0.

[0152] In an embodiment, the steps of the pooling calculation and the unpooling calculation specifically include:

[0153] Step 4.3: Graph pooling and graph unpooling:

[0154] Since the graph nodes do not have spatial locality required by the conventional pooling operation and do not carry spatial order information, the CNN-based pooling calculation cannot be used. Therefore, the graph pooling operation is used to sample the nodes in the present application. All node features are projected to one dimension using a trainable projection vector, and nodes with the largest scalar projection value are selected to form a new graph. At the same time, the present application retains the position information of the unselected nodes to restore the original graph structure in the graph unpooling.

[0155] In an embodiment, as shown in FIG. 1, the step of generating an updated blood vessel segmentation result according to the second graph structure includes:

[0156] The second graph structure is input into a graph convolutional network to obtain a third graph structure, which is compared with a benchmark ground truth to obtain a node binary cross-entropy loss. Iteration is performed until the node binary cross-entropy loss converges, and an updated blood vessel segmentation result is obtained.

[0157] In an embodiment, the node binary cross-entropy loss specifically includes:

[0158] Step 4.3: Loss function:

[0159] The application adopts node binary cross-entropy loss in the second stage network:

[0160] wherein |V| represents the number of graph nodes, t(v i ) and p(v i ) represent the true value and the predicted label of the graph node v i respectively.

[0161] Figure 6 shows a quantitative result comparison chart of selecting 9 segmentation models as a baseline on the MR data set. As can be seen from the comparison of various evaluation data, the method proposed in the application can achieve the effect of more accurately detecting high-difficulty blood vessels, and can better adapt to different data conditions and show the connectivity of blood vessels.

[0162] Figure 7 is a performance comparison visualization chart of UNETR++ as the first stage segmentation network on the MR data set according to an embodiment of the application, Figure 8 is a comparison visualization chart of the MR data set and other methods according to an embodiment of the application, and Figure 9 is a comparison visualization chart of the CT data set and other methods according to an embodiment of the application. The advantages of the application can be more intuitively shown from the various effect comparison charts.

[0163] Correspondingly, the embodiments of the application also provide a computer readable storage medium, wherein computer executable instructions are stored, and the computer executable instructions are executed by a processor to implement the method embodiments of the application. The computer readable storage medium includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, the computer readable storage medium does not include transitory computer readable media such as modulated data signals and carriers.

[0164] Further, the embodiments of the present application also provide a medical image based blood vessel segmentation device, which comprises a memory for storing computer executable instructions, and a processor; the processor is used to implement the steps in the above method embodiments when executing the computer executable instructions in the memory. The processor can be a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a microcontroller unit (MCU), a neural processing unit (NPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, etc. The memory can be a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk or a solid state disk, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware processor execution, or be executed by a combination of hardware and software modules in the processor.

[0165] Further, the embodiments of the present application also provide a computer program product, which comprises computer executable instructions, and the computer executable instructions are used to implement the steps in the above method embodiments when executed by a processor.

[0166] It is to be noted that, in the present application, the terms such as first and second, and the like, are used merely to differentiate one entity or action from another entity or action, and do not necessarily require or imply that there is any such actual relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a", "comprising a", or "comprises an" does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. In the present application, if an action is referred to as being performed "according to" a certain element, it means that the action is performed at least according to the element, including two cases: the action is performed only according to the element, and the action is performed according to the element and other elements. The expressions multiple, multiple times, multiple kinds, and the like, include 2, 2 times, 2 kinds, and more than 2, more than 2 times, more than 2 kinds.

[0167] The sequence numbers used in describing steps of a method do not in themselves imply any particular order between such steps. For instance, a step with a higher sequence number does not necessarily have to be performed after a step with a lower sequence number, but can also be performed before the step with the lower sequence number, or the steps can be performed in parallel, as long as such an execution order is reasonable for a person skilled in the art. As another example, multiple steps with consecutive numbering (e.g. step 101, step 102, step 103, etc.) do not limit that other steps can be performed in between, e.g. between step 101 and step 102 other steps can be performed.

[0168] The present specification includes the combination of various embodiments described herein. Individual references to embodiments, e.g. "an embodiment" or "some embodiments" or "a preferred embodiment"; do not necessarily refer to the same embodiments, although they can. Unless indicated to the contrary, the word "or" as used in this specification is intended to mean an inclusive "or", i.e. an "or" that does not exclude the presence of all of items or alternatives listed with "or". Further, unless otherwise indicated herein, the description as set forth is not intended to be limiting.

[0169] All documents mentioned in this specification are hereby incorporated by reference in their entirety for the disclosure they contain, to the extent that they do not conflict with the disclosure provided herein. Further, it is to be understood that the description of the present specification is merely exemplary of the principles of the present specification, and that numerous modifications can be made by those skilled in the art without departing from the scope of the present specification. Any and all modifications, variations or equivalents that fall within the spirit and scope of the principles of the present specification are intended to be included herein.

[0170] In some cases, the actions recited in the claims can be performed in a different order and still accomplish the desired result. In addition, the process depicted in the accompanying figures does not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

Claims

1. A method of blood vessel segmentation based on a medical image, characterized by, The method comprises the steps of: performing preliminary blood vessel segmentation on a three-dimensional medical image using a segmentation model to generate a preliminary blood vessel segmentation result; locating blood vessels according to the preliminary blood vessel segmentation result to generate a first graph structure; encoding the first graph structure using a double-flow encoder based on a graph channel attention network and a graph space attention network, and outputting the encoding result to a double-flow decoder based on the graph channel attention network and the graph space attention network through a graph convolution network to decode the encoding result to obtain a second graph structure; wherein the graph channel attention network focuses on the mutual influence between node feature channels in the graph structure, and the graph space attention network focuses on node information and its relationship in the spatial channel of the graph structure; and generating an updated blood vessel segmentation result according to the second graph structure.

2. The medical image-based blood vessel segmentation method according to claim 1, characterized in that, The step of generating the first graph structure further comprises the steps of: dividing the preliminary blood vessel segmentation result into a plurality of super-pixel regions through clustering of pixels, constructing graph nodes with the centers of the super-pixel regions, and constructing graph edges with the plurality of graph nodes; wherein the clustering is based on a cosine distance difference as a directional metric and a pixel gray value difference as a distance metric.

3. The method of claim 1, wherein, The graph channel attention network further comprises the steps of: inputting a graph node feature and applying a first multi-layer perceptron to perform calculation; compressing the spatial dimension of the input feature channel through average pooling and maximum pooling to obtain two feature descriptors, respectively, and performing matrix multiplication operation on the two feature descriptors and the graph node feature to calculate attention; applying a second multi-layer perceptron to the two feature descriptors obtained through the attention calculation, respectively, and adding the output results of the two parts element by element to obtain a graph channel attention feature. 4.The method of claim 1, wherein The step of obtaining the second graph structure uses a U-shaped network. The U-shaped network has an input side and an output side on two sides, respectively, the input side includes one or more double-flow encoders, the output side includes one or more double-flow decoders, the double-flow encoders and the double-flow decoders are symmetrically distributed in pairs, each pair of double-flow encoders and double-flow decoders is located in the same layer, and the bottom is the graph convolution network; In the U-shaped network, the first graph structure is the input of the top-layer double-flow encoder, the input side is connected in the direction from the top layer to the bottom layer, the output side is connected in the direction from the bottom layer to the top layer, and the same layer is connected in the direction from the double-flow encoder to the double-flow decoder, and the output result of the top-layer double-flow decoder is the second graph structure.

5. The method of claim 4, wherein the step of segmenting the blood vessels is performed by using a method of segmenting blood vessels based on a medical image. The step of obtaining the second graph structure further comprises the steps of: inputting the first graph structure into the top-layer double-flow encoder to generate a first graph space attention feature in the graph space attention network and a first graph channel attention feature in the graph channel attention network, respectively; inputting the first graph space attention feature and the first graph channel attention feature into a feature fusion network to generate a feature fusion result and to splice to obtain a first encoding feature; applying a multi-layer perceptron to the first encoding feature to obtain a second encoding feature, performing matrix multiplication calculation on the first encoding feature and the second encoding feature to calculate attention, and performing graph pooling calculation to obtain a top-layer input graph structure; After passing through N-1 double-flow decoders, a bottom-layer input graph structure is obtained, which is applied to the graph convolution network to obtain a bottom-layer output graph structure; The bottom-layer output graph structure is input into an Nth double-flow decoder, and the bottom-layer output graph structure is subjected to graph reverse pooling calculation and then input into the graph spatial attention network together with the feature fusion result to generate a second graph spatial attention feature and into the graph channel attention network to generate a second graph channel attention feature; The second graph spatial attention feature and the second graph channel attention feature are spliced to obtain a first decoding feature, and a multi-layer perception is applied to the first decoding feature to obtain a second decoding feature, and the first decoding feature and the second decoding feature are subjected to matrix multiplication to obtain an N-1 layer output graph structure. After passing through N-1 double-flow decoders, a second graph structure is obtained, where N is a positive integer. The step of generating the feature fusion result further includes:

6. The method of claim 5, wherein the step of segmenting the blood vessels is performed by using a method of segmenting blood vessels from a medical image. The graph spatial attention feature is input into a first forward mapping function, a first convolution combination, a first backward mapping function, and subjected to first curve enhancement to obtain a first spatial feature, and the graph spatial attention feature is input into a second forward mapping function, a second convolution combination, a second backward mapping function, and subjected to second curve enhancement to obtain a second spatial feature; A multi-layer perception and third curve enhancement are applied to the graph channel attention feature to obtain a channel feature; The channel feature is spliced with the first spatial feature and the second spatial feature, respectively, and matrix multiplication is performed on the two results to obtain the feature fusion result. The graph pooling calculation and the graph reverse pooling calculation include:

7. The method of claim 5, wherein the step of segmenting the blood vessel is performed by using a method of segmenting a blood vessel based on a medical image. The graph pooling calculation includes sampling graph nodes using a graph pooling operation, projecting all node features to one dimension using a trainable projection vector, and selecting nodes with the largest scalar projection value to form a new graph while retaining the position information of unselected nodes; and The graph reverse pooling calculation includes restoring the original graph structure using the result of the graph pooling calculation and the position information of the nodes. It includes:

8. A medical image-based blood vessel segmentation apparatus, characterized by comprising: a memory for storing computer executable instructions; and a processor coupled to the memory for implementing steps in the method of any one of claims 1 to 7 when executing the computer executable instructions. The computer readable storage medium stores computer executable instructions, and the computer executable instructions are executed by the processor to implement steps in the method of any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer executable instructions are executed by the processor to implement steps in the method of any one of claims 1 to 7.

10. A computer program product comprising computer executable instructions, characterised in that, ​

Images