Method and apparatus for blood vessel segmentation by link prediction with graph neural networks

Abstract

The application provides a blood vessel segmentation method and device through link prediction of a graph neural network. The blood vessel segmentation method through link prediction of the graph neural network comprises the following steps: inputting a to-be-detected blood vessel image into a U-net model, and obtaining a target first adjacency matrix output by the U-net model; and generating a target blood vessel connected graph based on a node set, an edge set and the target first adjacency matrix of the to-be-detected blood vessel image. The blood vessel segmentation method through link prediction of the graph neural network can generate a target first adjacency matrix with better connectivity and expansibility based on a to-be-detected blood vessel image through an optimized U-net model, and then generate a target blood vessel connected graph based on a node set, an edge set and the target first adjacency matrix, so that the topological structure and connectivity of the blood vessel can be significantly enhanced, and the segmentation efficiency and effect of the blood vessel image can be improved.

Description

Technical Field

[0001] This invention relates to the field of blood vessel image segmentation technology, and in particular to a method and apparatus for blood vessel segmentation using link prediction via graph neural networks. Background Technology

[0002] Blood vessel image segmentation technology is widely used in the diagnosis of coronary artery abnormalities. Graph Neural Networks (GNNs) have been introduced into the field of medical image segmentation. GNN-based models typically divide the image into non-overlapping sub-regions and sample some pixels from them to construct graph nodes and their features. However, this graph construction method generates many redundant and isolated nodes, thereby increasing the computational cost in the graph construction and training process. Furthermore, the final segmentation results do not show good performance in terms of the topological structure and connectivity of the blood vessel segmentation results. Summary of the Invention

[0003] This invention provides a method and apparatus for blood vessel segmentation using link prediction via graph neural networks, which addresses the shortcomings of existing technologies in terms of poor topological structure and connectivity of blood vessel segmentation results, thereby achieving efficient blood vessel image segmentation.

[0004] This invention provides a method for blood vessel segmentation using link prediction via a graph neural network, comprising:

[0005] The blood vessel image to be tested is input into the U-net model to obtain the target first adjacency matrix output by the U-net model;

[0006] Based on the node set, edge set, and target first adjacency matrix of the blood vessel image to be tested, a target blood vessel connectivity graph is generated;

[0007] The U-net model is trained using sample blood vessel images with edge labels as training samples and based on a target loss function.

[0008] According to the present invention, a method for blood vessel segmentation using link prediction via a graph neural network includes, prior to inputting the blood vessel image to be tested into the U-net model, the method comprising:

[0009] Based on the node set and edge set of the sample blood vessel image, generate the first adjacency matrix of the sample;

[0010] Based on node features and the first adjacency matrix of the sample, predict the links between nodes and generate a second adjacency matrix;

[0011] The target loss function is determined based on the first adjacency matrix and the second adjacency matrix of the samples.

[0012] According to the present invention, a method for blood vessel segmentation using link prediction via a graph neural network is provided, wherein generating a first adjacency matrix of the samples based on the node set and edge set of the sample blood vessel image includes:

[0013] Cluster the target nodes in the node set to generate clusters corresponding to the target nodes;

[0014] Based on the cluster corresponding to the target node and the features of each pixel in the cluster, the node features corresponding to the target node are generated;

[0015] The first adjacency matrix of the sample is generated based on the cosine similarity between the node features.

[0016] According to the present invention, a method for blood vessel segmentation by link prediction using a graph neural network, wherein the method predicts links between nodes based on node features and a first adjacency matrix of the samples, and generates a second adjacency matrix, includes:

[0017] The node features are encoded based on graph convolution to generate node embedding representations;

[0018] The node embedding representation is decoded using the inner product to generate the second adjacency matrix.

[0019] According to the present invention, a method for blood vessel segmentation using link prediction via a graph neural network, wherein decoding the node embedding representation using an inner product to generate a second adjacency matrix includes:

[0020] Based on the formula:

[0021] withZ = Encoder(f, A)

[0022] Generate the second adjacency matrix, where, Let Z be the second adjacency matrix, f be the node embedding representation, A be the first adjacency matrix of the sample, Encoder be the convolutional layer of the target graph, and σ be the sigmoid function.

[0023] According to the present invention, a method for blood vessel segmentation using link prediction via a graph neural network, prior to generating a first adjacency matrix of the samples based on the node set and edge set of the sample blood vessel image, the method includes:

[0024] Based on the manual annotations of blood vessels in the sample blood vessel images, the skeleton structure of the blood vessels is generated;

[0025] Corner point sampling is performed on the skeleton structure to generate the node set; the node set includes corner points, and the Euclidean distance between each corner point is not less than a first target threshold.

[0026] Based on the artificial annotations of the blood vessels, the geodesic distance between two nodes in the node set is generated;

[0027] If the geodesic distance is less than the second target threshold, an edge is constructed between the nodes corresponding to the geodesic distance to generate the edge set;

[0028] The positive or negative value of the target edge in the edge set is determined based on the distance between the skeleton and the edge set.

[0029] The present invention also provides an apparatus for segmenting blood vessels through link prediction using a graph neural network, comprising:

[0030] The first processing module is used to input the blood vessel image to be tested into the U-net model and obtain the target first adjacency matrix output by the U-net model;

[0031] The second processing module is used to generate a target blood vessel connectivity graph based on the node set, edge set, and the target first adjacency matrix of the blood vessel image to be tested.

[0032] The U-net model is trained using sample blood vessel images with edge labels as training samples and based on a target loss function.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for blood vessel segmentation by link prediction via graph neural networks as described above.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for blood vessel segmentation by link prediction via graph neural networks as described above.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for blood vessel segmentation by link prediction via a graph neural network as described above.

[0036] The present invention provides a method and apparatus for blood vessel segmentation using link prediction via graph neural networks. By generating a target first adjacency matrix with better connectivity and extensibility based on the blood vessel image under test using an optimized U-net model, the target blood vessel connectivity graph is generated based on the node set, edge set, and target first adjacency matrix. This method can significantly enhance the topology and connectivity of blood vessels, thereby improving the segmentation efficiency and effect of blood vessel images. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is one of the flowcharts illustrating the method for blood vessel segmentation using link prediction via graph neural networks provided by the present invention.

[0039] Figure 2 This is the second flowchart of the method for blood vessel segmentation using link prediction via graph neural networks provided by the present invention.

[0040] Figure 3 This is one of the schematic diagrams illustrating the effect of the method for blood vessel segmentation using link prediction via graph neural networks provided by the present invention;

[0041] Figure 4 This is the second schematic diagram illustrating the effect of the method for blood vessel segmentation using link prediction via graph neural networks provided by the present invention;

[0042] Figure 5 This is a schematic diagram of the device for blood vessel segmentation using link prediction via graph neural network provided by the present invention.

[0043] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0045] The following is combined Figures 1 to 4 The present invention describes a method for blood vessel segmentation using link prediction via a graph neural network.

[0046] It should be noted that the execution subject of the method for blood vessel segmentation by link prediction through graph neural network of the present invention can be a device for blood vessel segmentation by link prediction through graph neural network, a server, or a user terminal, such as a mobile phone, tablet computer or PC.

[0047] like Figure 1As shown, the method for blood vessel segmentation through link prediction using a graph neural network includes steps 110 and 120.

[0048] Step 110: Input the blood vessel image to be tested into the U-net model and obtain the target first adjacency matrix output by the U-net model;

[0049] In this step, the image of the blood vessel to be tested is the raw image acquired by the sensor, such as... Figure 4 As shown in (a).

[0050] The target first adjacency matrix is ​​a two-dimensional array used to construct the image, storing the relationships between nodes.

[0051] The target first adjacency matrix is ​​used to characterize the attributes of each edge, such as the weight corresponding to each edge.

[0052] The target's first adjacency matrix corresponds to good topology.

[0053] In practical applications, the U-net model is an optimized model used to generate a target first adjacency matrix with good blood vessel connectivity based on the blood vessel image to be tested.

[0054] The U-net model is trained using sample blood vessel images with edge labels as training samples, based on a target loss function.

[0055] In actual training, each blood vessel image with edge labels can be used as a training sample, thus obtaining multiple training samples to form a sample dataset. A portion of the training samples in the sample dataset is used as the training set, a portion as the validation set, and the remaining training samples as the test set. The ratio of training samples in the training set, validation set, and test set can be set to 5:2:3.

[0056] For example, the sample dataset could include the DRIVE dataset, which is the most common benchmark for fundus images, consisting of 40 fundus images of size 565×584 with human annotations.

[0057] The sample dataset may also include a dataset of coronary artery DSA images, which consists of 1092 coronary artery DSA images of size 512×512 with manual annotations. Of these, 546 images are divided into the training set, 218 images into the validation set, and the remaining 328 images constitute the test set.

[0058] The target loss functions include: the segmentation loss function and the graph constraint connection loss function.

[0059] In the actual training process, the initial U-net model can be trained first using the segmentation loss function.

[0060] The initial U-net model takes labeled sample blood vessel images as input and outputs the first adjacency matrix of the samples corresponding to the labeled sample blood vessel images.

[0061] The segmentation loss function is generated based on the Dice loss function and the cross-entropy (CE) loss function.

[0062] Specifically:

[0063]

[0064] Among them, L seg For the segmentation loss function; N p p is the number of pixels. i Let y be the image corresponding to the first adjacency matrix of pixel i output by the initial U-net model; i GT is for manual annotation; ε is the smoothing term, which can be set to 1e-6.

[0065] Then the LP-GCCM model was used to optimize the initial U-net model.

[0066] The LP-GCCM model takes as input the image the first adjacency matrix of the sample with edge labels, and outputs as the second adjacency matrix.

[0067] The image corresponding to the first adjacency matrix of the sample is generated by the U-net model before optimization (i.e., the initial U-net model).

[0068] For the LP-GCCM model, CE with a sigmoid layer can be used as the graph constraint connectivity loss L. GCC ;in:

[0069]

[0070] Where, N e p is the number of pixels. i The probability output for edge i (i.e., the image corresponding to the first adjacency matrix of the sample); y i Let i be the edge label corresponding to edge i.

[0071] The objective loss function of the entire optimized U-net model can then be expressed as:

[0072] L sum =L Seg +L GCC

[0073] In other words, during the actual training process, the initial U-net model is tuned based on the first and second adjacency matrices of the samples as the target loss function to obtain the optimized U-net model.

[0074] like Figure 2 As shown, for a sample blood vessel image, it is first input into the initial U-net model for blood vessel segmentation, generating the predicted image and node features corresponding to the first adjacency matrix of the sample; then, the predicted image and node features corresponding to the first adjacency matrix of the sample are input into the LP-GCCM model, which outputs the second adjacency matrix; then, the initial U-net model is adjusted based on the first and second adjacency matrices of the sample to enhance the blood vessel connectivity corresponding to the first adjacency matrix of its output, resulting in the optimized U-net model.

[0075] Step 120: Generate a connected graph of the target blood vessel based on the node set, edge set, and first adjacency matrix of the image of the blood vessel to be tested;

[0076] In this step, the node set is the set of at least some nodes on the blood vessel in the image of the blood vessel to be tested, such as... Figure 3 As shown in (c).

[0077] The edge set is the set of edges formed by at least some of the adjacent nodes in the node set corresponding to the blood vessel image to be tested, such as... Figure 3 As shown in (e).

[0078] The target vessel connectivity map is the final image generated after segmenting the original image (i.e., the vessel image to be tested).

[0079] The target vessel connectivity map can be described as follows:

[0080] G = (V; E; A)

[0081] Where G is the target blood vessel connected graph, V is the node set, E is the edge set, and A is the target first adjacency matrix.

[0082] After generating the first adjacency matrix of the target through step 110, the target blood vessel connectivity map can be generated using conventional image generation methods, which will not be elaborated upon in this invention.

[0083] Table 1 illustrates the vascular connectivity indices of images generated by the method of the present invention and other related techniques. The baseline-based method achieves Dice scores of 0.8149 and 0.8870 on the DRIVE and coronary DSA datasets, respectively. The method proposed in this invention achieves Dice scores of 0.8267 and 0.8921 on the two datasets, respectively, which is superior to other existing related methods.

[0084] In addition, this invention can effectively improve the Dice and clDice indices through integrated modeling. The Dice and clDice indices have clDice scores of 0.8267 and 0.9206 respectively, showing better vascular connectivity.

[0085] Figure 4 Examples of vessel segmentation results from the DRIVE and coronary artery DSA datasets are shown (the first three rows are the results from the DRIVE dataset, and the last three rows are the results from the coronary artery DSA image dataset). In these images, (a) is the original image, (b) is manually annotated, (c) is the baseline, (d), (e), and (f) are segmentation results generated by other methods, and (g) is the target vessel connectivity map generated by the method of this invention. Figure 4 It is evident that the target blood vessel connectivity map generated by the method of the present invention can better represent blood vessel connectivity.

[0086] Table 1

[0087]

[0088] The method for blood vessel segmentation using link prediction via graph neural networks provided in this embodiment of the invention generates a target first adjacency matrix with superior connectivity and extensibility based on the blood vessel image to be tested using an optimized U-net model. Then, a target blood vessel connectivity graph is generated based on the node set, edge set, and target first adjacency matrix. This method can significantly enhance the topology and connectivity of blood vessels, thereby improving the segmentation efficiency and effectiveness of blood vessel images.

[0089] The training process of the U-net model will be explained below through specific examples.

[0090] In some embodiments, prior to step 110, the method may further include:

[0091] Based on the node set and edge set of the sample blood vessel image, generate the first adjacency matrix of the sample;

[0092] Based on node features and the first adjacency matrix of the samples, predict the links between nodes and generate the second adjacency matrix;

[0093] The target loss function is determined based on the first and second adjacency matrices of the samples.

[0094] In this embodiment, the node set of the sample blood vessel image is the set of at least some nodes on the blood vessels in the sample blood vessel image.

[0095] The edge set is the set of edges formed by at least some of the adjacent nodes in the node set corresponding to the sample blood vessel image.

[0096] The first adjacency matrix of the sample was generated based on the unoptimized U-net model (i.e., the initial U-net model). The blood vessel connectivity corresponding to the first adjacency matrix of the sample is relatively poor compared to the blood vessel connectivity corresponding to the first adjacency matrix generated by the optimized U-net model.

[0097] In actual execution, the relationships between nodes in the node set can be determined based on the existing node set and edge set; based on the relationships between nodes, the first adjacency matrix of the sample can be generated.

[0098] In some embodiments, generating a first adjacency matrix for the samples based on the node set and edge set of the sample blood vessel image may include:

[0099] Cluster the target nodes in the node set to generate clusters corresponding to the target nodes;

[0100] Based on the cluster corresponding to the target node and the features of each pixel in the cluster, generate the node features corresponding to the target node;

[0101] The first adjacency matrix of the samples is generated based on the cosine similarity between node features.

[0102] In this embodiment, the target node is any node in the node set corresponding to the sample blood vessel image.

[0103] The first adjacency matrix of the sample is a matrix that includes the cosine similarity between any two nodes.

[0104] In practice, the improved Simple Linear Iterative Cluster (SLIC) method can be used to cluster each node simultaneously, using the vector distance between any two pixels to represent the similarity between pixels; the larger the distance, the smaller the similarity.

[0105] Specifically, during the iteration process of the SLIC algorithm, the target node v is fixed. j Position coordinates (x) i y i ) serves as the cluster center, thereby generating clusters with each v i The corresponding cluster C i ; where v i Represents the i-th node in the node set, v i ∈V, where V is the set of nodes.

[0106] Then, the average value of the features of all pixels in the cluster corresponding to the target node is calculated as the node feature corresponding to the target node.

[0107] For example, it can be based on the formula:

[0108]

[0109] Determine the node characteristics corresponding to the target node, where f i C represents the node feature corresponding to the i-th node; i For the cluster corresponding to the i-th node; |C i |For Ci The number of pixels in (cx) j cy j ) is C i The coordinates of the j-th pixel in the image are (cx) j cy j ) is C i The features of the j-th pixel.

[0110] It should be noted that, in this embodiment, f i It is a one-dimensional vector of length 64.

[0111] After generating the node features corresponding to the target node, the first adjacency matrix of the sample can be generated by calculating the similarity between any two nodes.

[0112] Taking cosine similarity as an example, it can be expressed by the formula:

[0113]

[0114] Generate the first adjacency matrix of the samples, where A i，j e is the cosine similarity between the i-th node and the j-th node; i，j The edge constructed for the i-th node and the j-th node; E is the edge set; f i f represents the node feature corresponding to the i-th node; j Ner(v) represents the node feature corresponding to the j-th node. i ) represents the i-th node v i The set of adjacent nodes.

[0115] Based on A i，j This will generate the first adjacency matrix of the sample.

[0116] Of course, in other embodiments, other methods can be used to determine the similarity between any two nodes, and this invention does not limit this.

[0117] After obtaining the first adjacency matrix of the sample, the graph connection constraint module can be used to optimize the first adjacency matrix, predict the links between nodes, and generate the second adjacency matrix.

[0118] The graph connectivity constraint module includes graph neural networks (GNNs) and convolutional neural networks (CNNs).

[0119] Among them, GNN improves segmentation performance by aggregating node features in non-Euclidean domains to discover relationships between connected nodes.

[0120] CNN is a feedforward neural network that includes convolutional computations and has a deep structure, and can be used for supervised learning.

[0121] For example, a plug-in approach can be used to combine GNN with CNN, allowing GNN to constrain blood vessel connectivity only during the training phase, in order to optimize the training of the first adjacency matrix of the samples and generate the second adjacency matrix.

[0122] It should be noted that in this invention, edge labels are selectively generated, and link prediction is performed based on these edge labels; rather than treating all existing edges as positive edges for link prediction, in order to effectively improve the topology and connectivity of blood vessels. The process of determining edge labels will be described in subsequent embodiments and will not be elaborated here.

[0123] In this step, the link prediction task demonstrates excellent ability to maintain vessel segmentation connectivity.

[0124] In some embodiments, based on node features and a sample first adjacency matrix, predicting links between nodes and generating a second adjacency matrix may include:

[0125] Node features are encoded using graph convolution to generate node embedding representations;

[0126] The inner product is used to decode the embedding representation to generate a second adjacency matrix.

[0127] In this embodiment, vascular connectivity can be enhanced using an LP-GCCM network to generate a second adjacency matrix.

[0128] A GNN is used to encode node features through graph convolution to obtain the node embedding representation corresponding to each node feature, and the inner product is used to decode the node embedding representation to reconstruct the adjacency matrix and generate the second adjacency matrix.

[0129] In some embodiments, decoding the embedding representation using the inner product to generate a second adjacency matrix may include:

[0130] Based on the formula:

[0131] with Z = Encoder(f, A)

[0132] Generate a second adjacency matrix, where, Let Z be the second adjacency matrix, f be the node embedding representation, A be the first adjacency matrix of the sample, Encoder be the convolutional layer of the target graph, and σ be the sigmoid function.

[0133] In this embodiment, the target graph convolutional layer can be an arbitrary graph convolutional layer.

[0134] In Variational Graph Normalized Autoencoder (VGNAE), the encoder is a graph normalized convolutional network (GNGN) using L2 normalization.

[0135] During the research and development process, the inventors discovered that the traditional U-net is widely used in medical image segmentation and has become the baseline for many segmentation tasks. Among related technologies, there are U-net variants that improve the traditional U-net network architecture or training strategies to achieve better segmentation performance, such as the nnU-net model. As an out-of-the-box tool based on U-net, it can generate state-of-the-art segmentation results in many medical segmentation tasks without human intervention. However, these CNN-based methods are mostly trained with uniformly weighted pixel loss, which often ignores the relatively weak connections between blood vessels, disrupting the topology and connectivity of vessel segmentation.

[0136] During the research and development process, the inventors also discovered that in related technologies, there are GNN-based medical image segmentation techniques. GNN-based models typically segment the image into non-overlapping sub-regions and sample some pixels from these regions to construct graph nodes and their features. If two nodes are both on blood vessels and are close to each other, they are connected by edges. However, on the one hand, this graph construction method generates many redundant and isolated nodes, which significantly increase the computational cost of graph construction and training. On the other hand, this method only uses GNN as a downstream task for node classification, aiming to label sub-regions as blood vessels or non-blood vessels, offering little help to the topological structure and connectivity of the blood vessel segmentation results.

[0137] In this invention, a Generative Neural Network (GNN) based on vascular topology is combined with a classic CNN model for vascular segmentation via a plug-in approach. First, U-net is used as the base segmentation model to generate segmentation results (i.e., the first adjacency matrix of samples) and node features for graph construction. Then, semantic information from U-net is used as node features to extend the Graph Connection Constraint Module (GCCM) to predict connections between nodes, thereby achieving link prediction and maintaining the vascular topology. Experiments show that the method of this invention can more effectively utilize semantic information from the CNN model, significantly reducing model training time; based on link prediction, it can also effectively maintain the connectivity of vascular segmentation.

[0138] In this invention, the method of integrating the blood vessel segmentation based on the basic U-net and the blood vessel segmentation based on the link prediction graph neural network module is used to take the union of the two methods. This avoids the GNN from damaging the performance of the basic segmentation model U-net due to the catastrophic forgetting phenomenon, thereby improving the overall segmentation accuracy and blood vessel connectivity.

[0139] The method for blood vessel segmentation based on link prediction using graph neural networks provided in this embodiment of the invention integrates blood vessel segmentation based on the basic U-net and blood vessel segmentation based on the link prediction graph neural network module by taking the union of the two methods. This avoids the GNN from damaging the performance of the basic segmentation model U-net due to catastrophic forgetting, thereby improving the overall segmentation accuracy and blood vessel connectivity.

[0140] In some embodiments, before generating the first adjacency matrix of the samples based on the node set and edge set of the sample blood vessel image, the method may further include:

[0141] Based on the manual annotation of blood vessels in sample blood vessel images, the skeleton structure of blood vessels is generated.

[0142] Corner sampling is performed on the skeleton structure to generate a node set; the node set includes corner points, and the Euclidean distance between each corner point is not less than the first target threshold.

[0143] Based on the artificial annotation of blood vessels, the geodesic distance between two nodes in the node set is generated;

[0144] If the geodesic distance is less than the second target threshold, an edge is constructed between the nodes corresponding to the geodesic distance to generate an edge set;

[0145] The positive or negative value of the target edge in the edge set is determined based on the distance between the skeleton and the edge set.

[0146] In this embodiment, corner points are pixels in the image, and corner points can be used as nodes in the graph.

[0147] Artificial annotations (GT) of blood vessels are artificially annotated basic structures of blood vessels.

[0148] The target edge is any edge in the edge set.

[0149] The positive or negative value of the target edge is used to determine the edge label.

[0150] In actual implementation, given artificial annotations of blood vessels, such as Figure 3 As shown in (a); and morphological methods were applied to obtain the artificially annotated skeleton structure I. S ,like Figure 3 As shown in (b); then in the skeleton structure I S Corner sampling is performed, and the sampled corners are used as nodes to generate a node set.

[0151] It should be noted that during the corner sampling process, the Euclidean distance between corner points should not be lower than the first target threshold.

[0152] The first target threshold can be user-defined, such as setting the first target threshold to 5 to ensure that the corner points are distributed on the blood vessels.

[0153] For example, the formula can be applied:

[0154]

[0155] Generate a node set, where V is the node set; v i For the i-th node; N n The number of corner points; (x i y i ) represents the x and y coordinates of the i-th node.

[0156] In some embodiments, N can be n The upper limit is set to 1000.

[0157] Of course, in other embodiments, N can also be... n The upper limit is set to other values, and the first target threshold is set to other values. This invention does not limit these values.

[0158] During the research and development process, the inventors discovered that traditional corner point acquisition methods extract pixels from a fixed-size sub-region, which is inefficient. For example, using traditional acquisition methods, a total of 20,732 nodes were sampled from the 21st image in the DRIVE dataset. However, only 3,380 of these nodes had edges between them, and approximately 6,366 nodes were in a black background, resulting in low sampling efficiency and poor sampling quality.

[0159] In this invention, corner sampling is performed on the skeleton to ensure that the Euclidean distance between each corner is greater than the first target threshold, thereby generating a node set. This not only improves the sampling efficiency but also effectively ensures that the corners are distributed on the blood vessels, significantly improving the sampling effect.

[0160] After generating the node set, an edge construction method is used, which uses the geodesic true distance as the standard for edge construction to build the edge set.

[0161] During the construction process, the travel time T from the target node in node set V to any other node in the node set can be calculated using the fast travel method. For the target node v i Travel time can be calculated using the following formula:

[0162]

[0163] in, For the i-th node v i The travel time T to any other node in the node set V; S is the velocity function, also known as the manual annotation, which characterizes the faster movement speed of a node on a blood vessel.

[0164] Calculate the travel time T(v) between each node and other nodes. iThe travel time is used as the geodesic distance, and an edge set is constructed between two nodes whose geodesic distance is less than the second target threshold. Specifically, this can be achieved using the formula:

[0165]

[0166] Construct an edge set, where E is the edge set; e i，j For node v i to node v j Directed edge; T(v i v j ) is node v i and node v j Travel time between; tt The second target threshold; N e Let be the number of sides.

[0167] The second target threshold can be user-defined.

[0168] Edge labels are used to train link prediction.

[0169] Through the skeleton structure I S and any edge e in edge set E i，j The distance between them determines the sign of an edge, and edges with positive signs are marked as edge labels, where e i，j ∈E.

[0170] For the target edge e i，j In an image, lines can be used to represent it.

[0171] Calculate e i，j with I S The minimum distance, and the minimum distance is less than the third target threshold. dis The distance is determined to be positive, and the minimum distance is not less than the third target threshold. dis The distance is determined to be negative, such as... Figure 3 As shown in (f), the edge with a positive minimum distance is a positive edge, and the edge with a negative minimum distance is a negative edge. The positive edges are almost perpendicular to the skeleton structure I. S coincide.

[0172] It should be noted that in this invention, not all existing edges are treated as positive edges for link prediction. Instead, edges whose minimum distance to the skeleton structure is less than the third target threshold are identified as positive edges, and link prediction is performed based on the finally determined positive edges, thereby effectively improving the accuracy and precision of the prediction results.

[0173] The method for blood vessel segmentation by link prediction using graph neural networks provided in this embodiment of the invention performs skeletonization on labeled blood vessels, takes the corner points on the skeleton as nodes, and uses semantic information from U-net as node features; then constructs edges based on the real distance between graph nodes, and uses the cosine similarity between two nodes as the weight of the edges to generate a first adjacency matrix. This method can effectively utilize the semantic information from the CNN model and significantly reduce the model training time.

[0174] The apparatus for blood vessel segmentation by link prediction using graph neural networks provided by the present invention will be described below. The apparatus for blood vessel segmentation by link prediction using graph neural networks described below can be referred to in correspondence with the method for blood vessel segmentation by link prediction using graph neural networks described above.

[0175] like Figure 5 As shown, the device for segmenting blood vessels through link prediction using a graph neural network includes: a first processing module 510 and a second processing module 520.

[0176] The first processing module 510 is used to input the blood vessel image to be tested into the U-net model and obtain the target first adjacency matrix output by the U-net model.

[0177] The second processing module 520 is used to generate a target blood vessel connectivity graph based on the node set, edge set, and target first adjacency matrix of the blood vessel image to be tested.

[0178] The U-net model is trained using sample blood vessel images with edge labels as training samples, based on a target loss function.

[0179] The apparatus for segmenting blood vessels using link prediction via graph neural networks, provided in this embodiment of the invention, generates a target first adjacency matrix with superior connectivity and extensibility based on the blood vessel image to be tested using an optimized U-net model. Then, it generates a target blood vessel connectivity graph based on the node set, edge set, and target first adjacency matrix. This significantly enhances the topology and connectivity of blood vessels, thereby improving the segmentation efficiency and effectiveness of blood vessel images.

[0180] In some embodiments, the device may further include:

[0181] The third processing module is used to generate the first adjacency matrix of the sample based on the node set and edge set of the sample blood vessel image;

[0182] The fourth processing module is used to predict the links between nodes and generate the second adjacency matrix based on node features and the first adjacency matrix of the samples.

[0183] The fifth processing module is used to determine the target loss function based on the first and second adjacency matrices of the samples.

[0184] In some embodiments, the third processing module may also be used for:

[0185] Cluster the target nodes in the node set to generate clusters corresponding to the target nodes;

[0186] Based on the cluster corresponding to the target node and the features of each pixel in the cluster, generate the node features corresponding to the target node;

[0187] The first adjacency matrix of the samples is generated based on the cosine similarity between node features.

[0188] In some embodiments, the fourth processing module may also be used for:

[0189] Node features are encoded using graph convolution to generate node embedding representations;

[0190] The node embedding representation is decoded using the inner product to generate the second adjacency matrix.

[0191] In some embodiments, the fourth processing module may also be used for:

[0192] Based on the formula:

[0193] with Z = Encoder(f, A)

[0194] Generate a second adjacency matrix, where, Let Z be the second adjacency matrix, f be the node embedding representation, A be the first adjacency matrix, Encoder be the target graph convolutional layer, and σ be the sigmoid function.

[0195] In some embodiments, the device may further include a fifth processing module for:

[0196] Before generating the first adjacency matrix of the sample based on the node set and edge set of the sample blood vessel image, the skeleton structure of the blood vessel is generated based on the manual annotation of the blood vessels in the sample blood vessel image.

[0197] Corner sampling is performed on the skeleton structure to generate a node set; the node set includes corner points, and the Euclidean distance between each corner point is not less than the first target threshold.

[0198] Based on the artificial annotation of blood vessels, the geodesic distance between two nodes in the node set is generated;

[0199] If the geodesic distance is less than the second target threshold, an edge is constructed between the nodes corresponding to the geodesic distance to generate an edge set;

[0200] The positive or negative value of the target edge in the edge set is determined based on the distance between the skeleton and the edge set.

[0201] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a method for blood vessel segmentation through link prediction using a graph neural network. This method includes: inputting the blood vessel image to be tested into a U-net model to obtain a target first adjacency matrix output by the U-net model; generating a target blood vessel connectivity graph based on the node set, edge set, and target first adjacency matrix of the blood vessel image to be tested; wherein the U-net model is trained using sample blood vessel images with edge labels as training samples and based on a target loss function.

[0202] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the method for blood vessel segmentation by link prediction through graph neural networks provided by the above methods, the method comprising: inputting a blood vessel image to be tested into a U-net model, obtaining a target first adjacency matrix output by the U-net model; generating a target blood vessel connectivity graph based on the node set, edge set and target first adjacency matrix of the blood vessel image to be tested; wherein the U-net model is trained using sample blood vessel images with edge labels as training samples, based on a target loss function.

[0204] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the methods for blood vessel segmentation by link prediction via graph neural networks provided above. The method includes: inputting a blood vessel image to be tested into a U-net model to obtain a target first adjacency matrix output by the U-net model; generating a target blood vessel connectivity graph based on the node set, edge set, and target first adjacency matrix of the blood vessel image to be tested; wherein the U-net model is trained using sample blood vessel images with edge labels as training samples based on a target loss function.

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for blood vessel segmentation by link prediction of graph neural networks, characterized in that, The method comprises the following steps: inputting a to-be-tested blood vessel image into a U-net model to obtain a target first adjacency matrix output by the U-net model; generating a target blood vessel connected graph based on a node set and an edge set of the to-be-tested blood vessel image and the target first adjacency matrix; wherein the U-net model is trained based on a target loss function and with a sample blood vessel image with edge labels as a training sample; before the step of inputting the to-be-tested blood vessel image into the U-net model, the method comprises the following steps: generating a sample first adjacency matrix based on a node set and an edge set of the sample blood vessel image; predicting links between nodes based on node features and the sample first adjacency matrix to generate a second adjacency matrix; determining the target loss function based on the sample first adjacency matrix and the second adjacency matrix; the target loss function comprises a segmentation loss function and a graph constraint connection loss function; in an actual training process, the segmentation loss function is used to train an initial U-net model first; the input of the initial U-net model is the sample blood vessel image with edge labels, and the output is a sample first adjacency matrix corresponding to the sample blood vessel image with edge labels; wherein the segmentation loss function is generated based on a Dice loss function and a cross-entropy (CE) loss function: ； wherein, is a segmentation loss function; is the number of pixels, is the image corresponding to the sample first adjacency matrix of pixel i output by the initial U-net model; is the manual annotation GT; is a smoothing term, is set to 1e-6; then, an LP-GCCM model is used to optimize the initial U-net model; the input of the LP-GCCM model is an image corresponding to the sample first adjacency matrix, and the output is the second adjacency matrix; wherein the image corresponding to the sample first adjacency matrix is generated by the initial U-net model; For the LP-GCCM model, CE with sigmoid layer is used as graph constraint connection loss ; wherein: ； wherein, is the number of pixels; the target loss function of the optimized U-net model can be expressed as: ； the step of generating the sample first adjacency matrix based on the node set and the edge set of the sample blood vessel image comprises the following steps: performing clustering on a target node in the node set to generate a cluster corresponding to the target node; generating node features corresponding to the target node based on the cluster corresponding to the target node and features of each pixel point in the cluster; generating the sample first adjacency matrix based on cosine similarities between the node features; the step of predicting links between nodes based on the node features and the sample first adjacency matrix to generate the second adjacency matrix comprises the following steps: encoding the node features based on a graph convolution to generate a node embedding representation; decoding the node embedding representation based on an inner product to generate the second adjacency matrix; the step of decoding the node embedding representation based on the inner product to generate the second adjacency matrix comprises the following steps: based on the formula: ； generating the second adjacency matrix, wherein for the second adjacency matrix, for the node embedding representation, for the node features, for the sample first adjacency matrix, for a target graph convolutional layer, sigmoid function.

2. The method for blood vessel segmentation by link prediction of graph neural network according to claim 1, characterized in that, before the step of generating the sample first adjacency matrix based on the node set and the edge set of the sample blood vessel image, the method comprises the following steps: generating a skeleton structure of a blood vessel in the sample blood vessel image based on artificial annotation of the blood vessel; performing corner point sampling on the skeleton structure to generate the node set; the node set comprises corner points, and a Euclidean distance between each two of the corner points is not less than a first target threshold; generating a geodesic distance between two nodes in the node set based on the artificial annotation of the blood vessel; In a case where the geodesic distance is less than a second target threshold, an edge is constructed between nodes corresponding to the geodesic distance, and the edge set is generated; Based on the distance between the skeleton and the edge set, the positive and negative values of a target edge in the edge set are determined.

3. An apparatus for vessel segmentation by link prediction of graph neural networks, the apparatus configured to perform the method for vessel segmentation by link prediction of graph neural networks according to any one of claims 1-2, characterized in that, Comprise: The first processing module is used for inputting the to-be-measured blood vessel image into a U-net model, and acquiring a target first adjacency matrix output by the U-net model; The second processing module is used for generating a target blood vessel connected graph based on a node set, an edge set of the to-be-measured blood vessel image, and the target first adjacency matrix; The U-net model is trained based on a target loss function by taking a sample blood vessel image with an edge label as a training sample.

4. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method for blood vessel segmentation by link prediction of a graph neural network according to any one of claims 1 to 2.

5. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method for blood vessel segmentation by link prediction of a graph neural network according to any one of claims 1 to 2.

6. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method for blood vessel segmentation by link prediction of a graph neural network according to any one of claims 1 to 2.

Images