Lightweight retinal vessel segmentation method based on graph convolution network and partial convolution

Abstract

The application provides a lightweight retinal blood vessel segmentation method based on a graph convolution network and a partial convolution, comprising a lightweight retinal blood vessel segmentation network model based on a graph convolution network and a partial convolution with a feature encoder, a multi-scale feature fusioner, a feature decoder and a label predictor, retinal blood vessel segmentation network model training and parameter optimization, and automatic semantic segmentation of retinal blood vessel structure. A symmetrical encoder-decoder deep learning model is built, partial convolution is used instead of conventional convolution to reduce the computational complexity of the model, K nearest neighbor algorithm is used to convert the feature map into a graph structure, then graph convolution is used to extract the global features of the image, a multi-scale feature fusion module is introduced to better fuse the encoder-decoder features, and a new upsampling method is realized by using a neighboring block expansion layer instead of a bilinear interpolation algorithm, the generalization of the model is improved, and accurate segmentation of the retinal blood vessels is realized.

Description

Technical Field

[0001] This invention relates to the field of medical image semantic segmentation technology, specifically to a lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution. Background Technology

[0002] The fundus is the only part of the human body where arteries, veins, and capillaries can be directly observed. By observing the geometry of retinal vessels, such as their angles, diameters, and topological structure, it is possible to effectively diagnose diabetes, hypertension, and various eye diseases. Among these diseases, diabetes can lead to retinal vascular disease and neovascularization, even causing blindness; hypertension can cause abnormal vascular proliferation, resulting in symptoms such as aneurysms and cotton wool spots. Furthermore, individuals with high myopia may exhibit diffuse or patchy retinal atrophy. Therefore, in clinical research, analyzing visual information about retinal vessels is crucial for helping doctors diagnose diabetes, hypertension, and various eye diseases, and excellent retinal vessel segmentation methods can improve diagnostic efficiency and accuracy.

[0003] However, the retinal vessels in the fundus have a complex dendritic topology, making them prone to rupture when they are too thin. The optic disc region has higher brightness and lower contrast than other areas, increasing the difficulty of retinal vessel segmentation. Furthermore, insufficient or excessive lighting can also reduce image contrast, resulting in unclear retinal vessel boundaries. In summary, due to the limitations of retinal vessel imaging itself and interference factors such as light sources during the imaging process, the contrast of vessels is reduced, which may lead to the loss of vascular information or directly affect the topological structure of the vessels. All these factors significantly increase the difficulty of retinal vessel segmentation.

[0004] With the rapid development of convolutional neural networks in the field of deep learning and the emergence of various network optimization techniques, more and more researchers at home and abroad are focusing on achieving automatic segmentation of retinal vessels and assisting in the diagnosis of diseases. In recent years, most automatic retinal vessel segmentation algorithms are based on encoder-decoder structures and use convolutional neural networks to extract image vessel features, ultimately achieving good segmentation results. However, the inventors of this application have found through research that such methods are limited by the size of the convolution kernel, cannot model global information, and will cause spatial information loss during downsampling, resulting in unclear vessel boundaries or noise in the segmentation results. Summary of the Invention

[0005] To address the technical problems of existing encoder-decoder-based retinal vessel segmentation methods failing to resolve vessel breaks and unclear boundaries in segmented images, this invention provides a lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution. This method constructs a convolutional neural network model, uses partial convolution instead of conventional convolution, and employs graph convolution to extract local and global features of the image separately. A multi-scale feature fusion module is introduced to better fuse encoder-decoder features, and a neighboring block expansion layer is used to implement a novel upsampling method instead of bilinear interpolation to improve the model's generalization and achieve accurate retinal vessel segmentation.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolutions includes the following steps:

[0008] S1. Construction of a lightweight retinal vessel segmentation network model based on graph convolutional networks and partial convolutions:

[0009] S11. A lightweight retinal vessel segmentation network based on graph convolutional networks and partial convolutions includes a feature encoder, a multi-scale feature fusion unit, a feature decoder, and a label predictor. The feature encoder captures local features of the image through partial convolutions and global features through graph convolutions, and performs downsampling through max pooling to reduce image resolution while increasing the semantic information contained in the image. The multi-scale feature fusion unit performs weighted fusion of local and global features, suppresses noise information in the feature layer, and focuses on the feature information of blood vessels. The feature decoder gradually restores the resolution of the feature map to the original resolution through neighboring block expansion layers and convolutional layers. Figure 1 Sample size; the label predictor obtains the final segmentation probability map by convolution and activation function from the multi-channel feature map output by the feature decoder;

[0010] S12. The feature encoder includes five feature encoding layer groups, and the feature decoder includes three feature decoding layer groups and four neighboring block expansion layers. Each feature encoding layer group and each feature decoding layer group consists of two partially convolutional layers and one regular convolutional layer arranged sequentially. A neighboring block expansion layer is set between each stage of the feature decoder. The neighboring block expansion layer replaces bilinear interpolation to perform upsampling operations on the feature map, gradually restoring the feature map to the original image size. The multi-scale feature fusion unit includes two pooling layer groups, two convolutional layers, and one softmax weight transformation layer. Each pooling layer group consists of a max pooling layer and an average pooling layer. A convolutional layer is set after each pooling layer group. The label predictor consists of a class prediction layer and a softmax probability transformation layer. The softmax probability transformation layer converts the class prediction score into a probability distribution.

[0011] S2. Training and parameter optimization of the retinal vessel segmentation network model:

[0012] S21. Network parameter initialization: The Xavier method is used to initialize the parameters of the lightweight retinal vessel segmentation network model based on graph convolutional network and partial convolution built in step S1.

[0013] S22. Dataset Preparation: The original retinal blood vessel images, which are divided into training and validation sets and have pixel-level segmentation labels, are separated into RGB three-channel feature maps and weighted. Then, histogram equalization is used to make the histogram of the retinal image approximately uniformly distributed, thereby enhancing the image contrast. Then, Gamma transformation is performed to more effectively preserve the image brightness information. Finally, online data augmentation technology is used to augment the training image data samples in the training set, thereby completing the preprocessing of the acquired retinal blood vessel images.

[0014] S23. Dataset Training: The preprocessed training set image data with pixel-level segmentation labels is used to train the segmentation network model using the 5-fold cross-validation method.

[0015] S24. Input the color fundus image of the same retinal vessel section into the network, and generate the retinal vessel segmentation result through forward computation of the network;

[0016] S25. The combination of the classification cross-entropy loss function and the set similarity loss function is used as the objective optimization function of the segmentation network, defined as follows:

[0017]

[0018] L2(θ')=1-Jaccard(Y′ ij -Y ij )

[0019] L(θ')=λ1L1(θ')+λ2L2(θ')

[0020] Where θ' represents the classification network parameters, L1(θ') is the cross-entropy loss function, L2(θ') is the set similarity loss function, S is the number of image pixels, C is the number of pixel categories, and Y′ ij It is a separator label, Y ij It predicts the label, Jaccard(Y′) ij -Y ij ) represents the similarity between the segmentation label and the prediction result, which is the ratio of the size of the intersection to the size of the union of the two sets. L(θ') is the objective optimization function of the segmentation network. λ1 is the weight factor of the weighted summation of the cross-entropy loss function, and λ2 is the weight factor of the weighted summation of the set similarity loss function.

[0021] S26. The objective function L(θ') is optimized using the adaptive moment estimation gradient descent algorithm, and the parameters of the retinal vessel segmentation network model are updated using error backpropagation to obtain the optimal network model parameters θ. best ;

[0022] S3. Automatic semantic segmentation of retinal vascular structures:

[0023] S31. Use the learned optimal network model parameters θ best We constructed a lightweight automatic semantic segmentation network for retinal vascular color fundus images based on graph convolutional networks and partial convolutions.

[0024] S32. First, perform online data augmentation on the color fundus image. Then, weight and merge the RGB three-channel image into a single-channel image and divide it into 64×64 image blocks. Input the blocks into the feature encoder to extract feature maps of different scales.

[0025] S33. A feature map of the same size as the original image is obtained through the feature decoder, and then fed into the category prediction layer in the label predictor to obtain the pixel category prediction scores for the blood vessel region and the background region. Finally, the softmax probability transformation layer is used to convert the prediction scores into a probability distribution.

[0026] S34. Take the index of the component with the highest probability of each pixel as the pixel category label to obtain the final binary image of retinal blood vessel semantic segmentation.

[0027] Furthermore, in step S12, the kernel size of the two partial convolutional layers in each feature encoding layer group and feature decoding layer group is 3×3 with a stride of 1, and the kernel size of a regular convolutional layer is 1×1 with a stride of 1. The number of kernels in the first partial convolutional layer is only 1 / 4 of the number of feature channels obtained by the previous feature encoder / decoder, that is, only the first 1 / 4 of the feature input channels are subjected to regular convolution for spatial feature extraction, while the remaining channels remain unchanged. The function of the second convolutional layer is to promote the interaction between the channels that have passed through the convolutional layer and the unconvolutional layers. Information exchange between channels in the convolutional layers; the number of convolutional kernels in the five feature coding layer groups are 64, 16, 128, 32, 32, 256, 64, 64, 512, 128, 128, 1024 respectively; the number of convolutional kernels in the three feature decoding layer groups are 128, 128, 512, 64, 64, 256, 32, 32, 128 respectively; the convolutional kernel size of the two convolutional layers in the multi-scale feature fusion unit is 1×1, and the stride is 1; the number of convolutional kernels in the label predictor is 16 and 2 respectively.

[0028] Furthermore, in step S22, online data augmentation techniques are used on the training image data samples in the training set, including random horizontal and vertical flipping, and rotations of 45°, 90°, 135°, 180°, 225°, 270°, and 315°, to increase the number of training image data samples to 10 times the initial value.

[0029] Furthermore, the forward computation of the network in step S24 includes convolution operations, graph convolution operations, batch normalization, nonlinear activation, and probability value transformation.

[0030] Furthermore, in the convolution operation, the output feature map Z corresponding to any convolution kernel... i The following formula is used for calculation:

[0031]

[0032] Where f represents the nonlinear excitation function, b i Let W represent the bias term corresponding to the i-th convolutional kernel, r represent the input channel index, k represent the number of input channels, and W represent the input channel number. ir This represents the weight matrix of the r-th channel of the i-th convolutional kernel. It is a convolution loss operation, X r This represents the image of the r-th input channel.

[0033] Furthermore, in the graph convolution operation, the Laplacian matrix is ​​used to define graph convolution for an undirected graph. A represents the adjacency matrix, and D represents the diagonal matrix. L = ID -1 / 2 AD -1 / 2 Indicates to The normalized Laplace matrix L can be decomposed into L = UΛU T Where U is the eigenvector matrix, Λ = diag[λ1,...,λ n [ ] represents the eigenvalue matrix; the graph convolutional network introduces a first-order approximation of ChebNet, and the graph convolutional network iteratively aggregates information from neighboring nodes, regarding node v i The forward propagation process is as follows:

[0034]

[0035] Where σ(·) is a nonlinear activation function, Let A and W represent the renormalized adjacency matrices. (l) Let v represent the learnable transformation matrix of the l-th layer. j Represents node v i The neighboring nodes, This represents the node feature matrix of the l-th layer.

[0036] Furthermore, the nonlinear excitation uses a rectified linear unit (ReLU) as the feature map Z. i The nonlinear activation function is used to transform the feature map Z i Each value in the equation undergoes a non-linear transformation, and the rectified linear unit ReLU is defined as follows:

[0037] f(x) = max(0,x)

[0038] Here, max represents finding the maximum value, and x is an input value.

[0039] Furthermore, the probability value transformation uses the Softmax function to convert the category prediction scores into a probability distribution. The Softmax function is defined as follows:

[0040]

[0041] Among them, Y j It is the probability that a pixel belongs to the j-th class, O j O i K is the predicted score of a pixel in the j-th and i-th classes at the end of the segmentation network output, where K represents the number of classes.

[0042] Furthermore, the specific optimization process in step S26 is as follows:

[0043]

[0044] m t =β1m t-1 +(1-β1)g t

[0045]

[0046]

[0047]

[0048]

[0049] Where t represents the number of update steps, and θ is the network parameter that needs to be updated, corresponding to θ' in the objective function of step S25. For the loss function with parameter θ, g t For the target loss function The gradient obtained by differentiating with respect to θ, β1 is the first-order moment decay coefficient, β2 is the second-order moment decay coefficient, and m t v t The gradients g are respectively t The first and second moments, For m t bias correction, For v t The bias correction, where η represents the learning rate, is used to control the stride, and the initial value is set to 5e. -4 The cosine annealing algorithm is used to gradually decay the learning rate to 1e. -5 , It is a small positive number to prevent the denominator from being zero.

[0050] Compared with the prior art, the lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution provided by the present invention has the following advantages: (1) The present invention introduces a graph convolutional module between the feature encoder and the feature decoder to model the global context information of the image, thereby capturing the long-distance dependency relationship between retinal vessel pixels and thus alleviating the phenomenon of vessel breakage in the segmentation result; (2) The present invention uses partial convolution to replace the conventional convolution in the feature encoder and decoder. Partial convolution can reduce redundant information in the feature layer and reduce memory access of convolution operation in the feature extraction stage, thereby improving the training speed and inference speed of the model; (3) The present invention introduces a multi-scale feature fusion module at the skip connection, which is mainly used to fuse the global features of the graph convolutional layer and the local features of the partial convolutional layer to reduce noise information in the image features and retain the vessel information; (4) In the feature decoder upsampling stage, the present invention adopts the neighboring block expansion operation to improve the ability to fill missing and blank values ​​in the process of image restoration from small size to large size while enlarging the feature map, avoiding discontinuity in the retinal vessel segmentation result and thus improving the segmentation accuracy. Attached Figure Description

[0051] Figure 1This is a schematic diagram of a lightweight retinal vessel segmentation network model based on graph convolutional networks and partial convolution provided by the present invention.

[0052] Figure 2 This is a schematic diagram of the partial convolution (PConv) operation provided by the present invention.

[0053] Figure 3 This is a schematic diagram of the Multi-scale Feature Fusion Block (MFFB) provided by the present invention.

[0054] Figure 4 This is a schematic diagram of the operation process of the PatchExpand layer provided by the present invention.

[0055] Figure 5 This is a schematic diagram of the final result of image data preprocessing provided by the present invention. Detailed Implementation

[0056] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.

[0057] Please refer to Figures 1 to 5 As shown, this invention provides a lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution, comprising the following steps:

[0058] S1. Construction of a lightweight retinal vessel segmentation network model based on graph convolutional networks and partial convolutions:

[0059] S11. A lightweight retinal vessel segmentation network based on graph convolutional networks and partial convolutions includes a feature encoder, a multi-scale feature fusion unit, a feature decoder, and a label predictor. The feature encoder captures local features of the image through partial convolutions and global features through graph convolutions, and performs downsampling through max pooling to reduce image resolution while increasing the semantic information contained in the image. The multi-scale feature fusion unit performs weighted fusion of local and global features, suppresses noise information in the feature layer, and focuses on the feature information of blood vessels. The feature decoder gradually restores the resolution of the feature map to the original resolution through neighboring block expansion layers and convolutional layers. Figure 1 Sample size; the label predictor obtains the final segmentation probability map by convolution and activation function from the multi-channel feature map output by the feature decoder;

[0060] S12. The feature encoder includes five feature encoding layer groups, and the feature decoder includes three feature decoding layer groups and four neighboring block expansion layers. Each feature encoding layer group and each feature decoding layer group consists of two partially convolutional layers and one regular convolutional layer arranged sequentially. A neighboring block expansion layer (PatchExpand) is set between each stage of the feature decoder. The neighboring block expansion layer replaces bilinear interpolation to perform upsampling operations on the feature map, gradually restoring the feature map to the original image size. The multi-scale feature fusion unit (MFFB) includes two pooling layer groups, two convolutional layers, and one softmax weight transformation layer. Each pooling layer group consists of a max pooling layer and an average pooling layer. A convolutional layer is set after each pooling layer group. The label predictor consists of a class prediction layer and a softmax probability transformation layer. The softmax probability transformation layer converts the class prediction score into a probability distribution.

[0061] In a specific embodiment, in step S12, the kernel size of the two partial convolutional layers in each feature encoding layer group and feature decoding layer group is 3×3 with a stride of 1, and the kernel size of a regular convolutional layer is 1×1 with a stride of 1. The number of kernels in the first partial convolutional layer is only 1 / 4 of the number of feature channels obtained by the previous feature encoder and decoder, that is, only the first 1 / 4 of the feature input channels are subjected to regular convolution for spatial feature extraction, while the remaining channels remain unchanged. The function of the second convolutional layer is to promote the flow of channels after passing through the convolutional layer. Information exchange between channels that have not passed through convolutional layers; the number of convolutional kernels in the five feature coding layer groups are 64, 16, 128, 32, 32, 256, 64, 64, 512, 128, 128, 1024 respectively; the number of convolutional kernels in the three feature decoding layer groups are 128, 128, 512, 64, 64, 256, 32, 32, 128 respectively; the convolutional kernel size of the two convolutional layers in the multi-scale feature fusion unit is 1×1, and the stride is 1; the number of convolutional kernels in the label predictor is 16 and 2 respectively. The entire segmentation network input is a 1-channel input with a channel size of 64×64, which is obtained by weighted summation of the red, green, and blue channels of the original color image, with weights of 0.299, 0.587, and 0.144, respectively. The network output is a 2-channel input with a channel size of 64×64, representing two categories: blood vessel pixels and background pixels. Detailed model parameters are shown in Table 1 below.

[0062] Table 1. Parameters of the retinal vessel segmentation network model (Padding=1)

[0063]

[0064]

[0065] In Table 1 above, in order to ensure that the feature map size remains unchanged during the convolution process, Padding = 1 is set, that is, 0 is used to fill the area around the image during convolution.

[0066] S2. Training and parameter optimization of the retinal vessel segmentation network model:

[0067] S21. Network parameter initialization: The Xavier method is used to initialize the parameters of the lightweight retinal vessel segmentation network model based on graph convolutional network and partial convolution built in step S1.

[0068] S22. Dataset Preparation: As a specific implementation method, the inventors of this application obtained patient data with pixel-level segmentation labels for 88 cases. The specific datasets used included the DRIVE dataset from the Dutch Diabetic Retinopathy Screening Project, the STARE dataset from the University of California, San Diego, and the CHASEDB1 dataset from the left and right eyes of 14 schoolchildren. The DRIVE dataset contains 40 images, each 768×584 pixels; the first 20 images were used as the training set, and the last 20 as the test set. The STARE dataset contains 20 images of 605×700 pixels; the first 10 were used as the training set, and the last 10 as the test set. The CHASEDB1 dataset contains 28 color retinal vascular images of 999×960 pixels; the first 20 were used as the training set, and the last 8 as the test set. Each image was annotated by two independent experts. Because the contrast between blood vessels and background is high in the green channel image of retinal vessels, the original retinal vessels images, divided into training and validation sets and labeled with pixel-level segmentation, are separated into RGB three-channel feature maps and weighted. Then, histogram equalization is used to make the histogram of the retinal image approximately uniformly distributed, thereby enhancing the image contrast. Then, Gamma transformation is performed to more effectively preserve the image brightness information. Finally, online data augmentation techniques with random horizontal and vertical flips and rotations of 45°, 90°, 135°, 180°, 225°, 270°, and 315° are used to augment the training image data samples in the training set, increasing the number of training image data samples to 10 times the initial amount, thus completing the preprocessing of the acquired retinal vessels images. This invention divides the training images and labels into 64×64 pixel blocks, with each pixel block spaced 16 pixels apart.

[0069] S23. Dataset Training: The preprocessed training set image data with pixel-level segmentation labels is used to train the segmentation network model using 5-fold cross-validation.

[0070] S24. Input color fundus images of the same retinal vessel section into the network, and generate retinal vessel segmentation results through network forward computation. In a specific embodiment, the network forward computation includes convolution operations, graph convolution operations, batch normalization, nonlinear activation, and probability value transformation.

[0071] Convolution operation: In the convolution operation, the output feature map Z corresponding to any convolution kernel. i The following formula is used for calculation:

[0072]

[0073] Where f represents the nonlinear excitation function, b i Let W represent the bias term corresponding to the i-th convolutional kernel, r represent the input channel index, k represent the number of input channels, and W represent the input channel number. ir This represents the weight matrix of the r-th channel of the i-th convolutional kernel. It is a convolution loss operation, X r This represents the image of the r-th input channel.

[0074] Graph Convolution Operation: Graph Convolutional Networks (GCNs) are deep learning models designed for graph-structured data. The core idea of ​​GCNs is to extend convolution operations to graph-structured data, achieving feature updates for nodes by aggregating information from neighboring nodes. In the graph convolution operation described in this invention, each pixel of the encoder feature map is treated as a graph node. The K-nearest neighbor algorithm is used to find the three nearest neighbors of each node as edges of the graph. Then, the Laplacian matrix is ​​used to define the graph convolution. For an undirected graph... A represents the adjacency matrix, and D represents the diagonal matrix. L = ID -1 / 2 AD -1 / 2 Indicates to The normalized Laplace matrix L can be decomposed into L = UΛU T Where U is the eigenvector matrix, Λ = diag[λ1,...,λ n [ ] represents the eigenvalue matrix; the graph convolutional network introduces a first-order approximation of ChebNet (K=1), and the graph convolutional network iteratively aggregates information from neighboring nodes, regarding node v i The forward propagation process is as follows:

[0075]

[0076] Where σ(·) is a nonlinear activation function, Let A and W represent the renormalized adjacency matrices. (l) Let v represent the learnable transformation matrix of the l-th layer. jRepresents node v i The neighboring nodes, This represents the node feature matrix of the l-th layer.

[0077] Batch Normalization: Batch Normalization (BN) is a method that makes neural network training faster and more stable. It is used to calculate the mean and variance of each mini-batch and pull them back to a standard normal distribution with a mean of 0 and a variance of 1. The specific operation and calculation are existing techniques well known to those skilled in the art, so they will not be described in detail here.

[0078] Nonlinear excitation: The nonlinear excitation uses a rectified linear unit ReLU as the feature map Z. i The nonlinear activation function is used to transform the feature map Z i Each value in the equation undergoes a non-linear transformation, and the rectified linear unit ReLU is defined as follows:

[0079] f(x) = max(0,x)

[0080] Here, max represents finding the maximum value, and x is an input value.

[0081] Probability value transformation: The probability value transformation uses the Softmax function to convert the category prediction scores into a probability distribution. The Softmax function is defined as follows:

[0082]

[0083] Among them, Y j It is the probability that a pixel belongs to the j-th class, O j O i K is the predicted score of a pixel in the j-th and i-th classes at the end of the segmentation network output, where K represents the number of classes.

[0084] S25. The combination of the classification cross-entropy loss function and the set similarity loss function is used as the objective optimization function of the segmentation network, defined as follows:

[0085]

[0086] L2(θ')=1-Jaccard(Y′ ij -Y ij )

[0087] L(θ')=λ1L1(θ')+λ2L2(θ')

[0088] Where θ' represents the classification network parameters, L1(θ') is the cross-entropy loss function, L2(θ') is the set similarity loss function, S is the number of image pixels, C is the number of pixel categories, and Y′ij It is a separator label, Y ij It predicts the label, Jaccard(Y′) ij -Y ij ) represents the similarity between the segmentation label and the prediction result, which is the ratio of the intersection size to the union size of the two sets. L(θ') is the objective optimization function of the segmentation network. λ1 is the weight factor of the weighted sum of the cross-entropy loss function, and λ2 is the weight factor of the weighted sum of the set similarity loss function. Among them, the values ​​of λ1 and λ2 should not be less than 0, and are usually set to 0.5 based on experience.

[0089] S26. The Adaptive Moment Estimation (Adam) gradient descent algorithm is used to optimize the objective function L(θ'), and the parameters of the retinal vessel segmentation network model are updated using error backpropagation to obtain the optimal network model parameters θ. best As a specific implementation method, the optimization process is as follows:

[0090]

[0091] m t =β1m t-1 +(1-β1)g t

[0092]

[0093]

[0094]

[0095]

[0096] Where t represents the number of update steps, and θ is the network parameter that needs to be updated, corresponding to θ' in the objective function of step S25. For the loss function with parameter θ, g t For the target loss function The gradient obtained by differentiating with respect to θ, β1 is the first-order moment decay coefficient, β2 is the second-order moment decay coefficient, and m t v t The gradients g are respectively t The first and second moments, For m t bias correction, For v t The bias correction, where η represents the learning rate, is used to control the stride, and the initial value is set to 5e. -4 The cosine annealing algorithm is used to gradually decay the learning rate to 1e. -5 , It is a small positive number to prevent the denominator from being zero.

[0097] S3. Automatic semantic segmentation of retinal vascular structures:

[0098] S31. Use the learned optimal network model parameters θ best We constructed a lightweight automatic semantic segmentation network for retinal vascular color fundus images based on graph convolutional networks and partial convolutions.

[0099] S32. First, perform online data augmentation on the color fundus image. Then, weight and merge the RGB three-channel image into a single-channel image and divide it into 64×64 image blocks. Input the blocks into the feature encoder to extract feature maps of different scales.

[0100] S33. A feature map of the same size as the original image is obtained through the feature decoder, and then fed into the category prediction layer in the label predictor to obtain the pixel category prediction scores for the blood vessel region and the background region. Finally, the softmax probability transformation layer is used to convert the prediction scores into a probability distribution.

[0101] S34. Take the index of the component with the highest probability of each pixel as the pixel category label to obtain the final binary image of retinal blood vessel semantic segmentation.

[0102] Compared with the prior art, the lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution provided by the present invention has the following advantages: (1) The present invention introduces a graph convolutional module between the feature encoder and the feature decoder to model the global context information of the image, thereby capturing the long-distance dependency relationship between retinal vessel pixels and thus alleviating the phenomenon of vessel breakage in the segmentation result; (2) The present invention uses partial convolution to replace the conventional convolution in the feature encoder and decoder. Partial convolution can reduce redundant information in the feature layer and reduce memory access of convolution operation in the feature extraction stage, thereby improving the training speed and inference speed of the model; (3) The present invention introduces a multi-scale feature fusion module at the skip connection, which is mainly used to fuse the global features of the graph convolutional layer and the local features of the partial convolutional layer to reduce noise information in the image features and retain the vessel information; (4) In the feature decoder upsampling stage, the present invention adopts the neighboring block expansion operation to improve the ability to fill missing and blank values ​​in the process of image restoration from small size to large size while enlarging the feature map, avoiding discontinuity in the retinal vessel segmentation result and thus improving the segmentation accuracy.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A lightweight retinal vessel segmentation method based on graph convolutional network and partial convolution, characterized in that, Includes the following steps: S1. Construction of a lightweight retinal vessel segmentation network model based on graph convolutional networks and partial convolutions: S11. A lightweight retinal vessel segmentation network based on graph convolutional networks and partial convolutions includes a feature encoder, a multi-scale feature fusion unit, a feature decoder, and a label predictor. The feature encoder captures local features of the image through partial convolutions and global features through graph convolutions, and performs downsampling through max pooling to reduce image resolution while increasing the semantic information contained in the image. The multi-scale feature fusion unit performs weighted fusion of local and global features, suppresses noise information in the feature layer, and focuses on the feature information of blood vessels. The feature decoder gradually restores the resolution of the feature map to the same size as the original image through the neighbor block expansion layer and the convolutional layer; the label predictor obtains the final segmentation probability map by convolution and activation function from the multi-channel feature map output by the feature decoder. S12. The feature encoder includes five feature encoding layer groups, and the feature decoder includes three feature decoding layer groups and four neighboring block expansion layers. Each feature encoding layer group and each feature decoding layer group consists of two partially convolutional layers and one regular convolutional layer arranged sequentially. A neighboring block expansion layer is set between each stage of the feature decoder. The neighboring block expansion layer replaces bilinear interpolation to perform upsampling operations on the feature map, gradually restoring the feature map to the original image size. The multi-scale feature fusion unit includes two pooling layer groups, two convolutional layers, and one softmax weight transformation layer. Each pooling layer group consists of a max pooling layer and an average pooling layer. A convolutional layer is set after each pooling layer group. The label predictor consists of a class prediction layer and a softmax probability transformation layer. The softmax probability transformation layer converts the class prediction score into a probability distribution. S2. Training and parameter optimization of the retinal vessel segmentation network model: S21. Network parameter initialization: The Xavier method is used to initialize the parameters of the lightweight retinal vessel segmentation network model based on graph convolutional network and partial convolution built in step S1. S22. Dataset Preparation: The original retinal blood vessel images, which are divided into training and validation sets and have pixel-level segmentation labels, are separated into RGB three-channel feature maps and weighted. Then, histogram equalization is used to make the histogram of the retinal image approximately uniformly distributed, thereby enhancing the image contrast. Then, Gamma transformation is performed to more effectively preserve the image brightness information. Finally, online data augmentation technology is used to augment the training image data samples in the training set, thereby completing the preprocessing of the acquired retinal blood vessel images. S23. Dataset Training: The preprocessed training set image data with pixel-level segmentation labels is used to train the segmentation network model using the 5-fold cross-validation method. S24. Input the color fundus image of the same retinal vessel section into the network, and generate the retinal vessel segmentation result through forward computation of the network; S25. The combination of the classification cross-entropy loss function and the set similarity loss function is used as the objective optimization function of the segmentation network, defined as follows: in, To classify network parameters, Let cross-entropy be the loss function. Let be the set similarity loss function. The number of pixels in the image. Number of pixel categories It is a separator tag. It is a predicted label. The similarity between the segmentation label and the prediction result is the ratio of the intersection size to the union size of the two sets. To optimize the objective function of the segmentation network, The weighting factors are used to weight the sum of the cross-entropy loss function. The weighting factors are used to weight the summation of the set similarity loss function; S26. Optimize the objective function using the adaptive moment estimation gradient descent algorithm. The parameters of the retinal vessel segmentation network model are updated using backpropagation of errors to obtain the optimal network model parameters. ; S3. Automatic semantic segmentation of retinal vascular structures: S31. Use the learned optimal network model parameters We constructed a lightweight automatic semantic segmentation network for retinal vascular color fundus images based on graph convolutional networks and partial convolutions. S32. First, perform online data augmentation on the color fundus image. Then, weight and merge the RGB three-channel image into a single-channel image and divide it into 64×64 image blocks. Input the blocks into the feature encoder to extract feature maps of different scales. S33. A feature map of the same size as the original image is obtained through the feature decoder, and then fed into the category prediction layer in the label predictor to obtain the pixel category prediction scores for the blood vessel region and the background region. Finally, the softmax probability transformation layer is used to convert the prediction scores into a probability distribution. S34. Take the index of the component with the highest probability of each pixel as the pixel category label to obtain the final binary image of retinal blood vessel semantic segmentation.

2. The lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution as described in claim 1, characterized in that, In step S12, the kernel size of the two partial convolutional layers in each feature encoding layer group and feature decoding layer group is 3×3 with a stride of 1, and the kernel size of a regular convolutional layer is 1×1 with a stride of 1. The number of kernels in the first partial convolutional layer is only 1 / 4 of the number of feature channels obtained by the previous feature encoder / decoder, that is, only the first 1 / 4 of the feature input channels are subjected to regular convolution for spatial feature extraction, while the remaining channels remain unchanged. The role of the second convolutional layer is to promote the interaction between the channels that have passed through the convolutional layer and those that have not. Information exchange between channels of the convolutional layers; the number of convolutional kernels in the five feature coding layer groups are 64, 16, 128, 32, 32, 256, 64, 64, 512, 128, 128, 1024 respectively; the number of convolutional kernels in the three feature decoding layer groups are 128, 128, 512, 64, 64, 256, 32, 32, 128 respectively; the convolutional kernel size of the two convolutional layers in the multi-scale feature fusion unit is 1×1, and the stride is 1; the number of convolutional kernels in the label predictor is 16 and 2 respectively.

3. The lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution as described in claim 1, characterized in that, In step S22, online data augmentation techniques are used on the training image data samples in the training set, including random horizontal and vertical flipping, and rotations of 45°, 90°, 135°, 180°, 225°, 270°, and 315°, to increase the number of training image data samples to 10 times the initial value.

4. The lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution as described in claim 1, characterized in that, The forward computation of the network in step S24 includes convolution operations, graph convolution operations, batch normalization, nonlinear activation, and probability value transformation.

5. The lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution as described in claim 4, characterized in that, In the convolution operation, the output feature map corresponding to any convolution kernel The following formula is used for calculation: in, Represents a nonlinear excitation function. Indicates the first The bias terms corresponding to each convolution kernel Indicates the input channel index number. Indicates the number of input channels. Indicates the first The first convolutional kernel Channel weight matrix, It is a convolution loss operation. Indicates the first One input channel image.

6. The lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution as described in claim 4, characterized in that, In the graph convolution operation, each pixel of the encoder feature map is treated as a graph node. The K-nearest neighbor algorithm is used to find the three nearest neighbors of each node as edges of the graph. Then, the Laplacian matrix is ​​used to define the graph convolution. For an undirected graph... , Represents the adjacency matrix. Represents a diagonal matrix. ; Indicates to The Laplace matrix after normalization. Decomposed into ,in The eigenvector matrix, The eigenvalue matrix is ​​used; the graph convolutional network introduces a first-order approximation of ChebNet, and the graph convolutional network iteratively aggregates information from neighboring nodes, regarding nodes. The forward propagation process is as follows: in, It is a non-linear activation function. Represents the renormalized adjacency matrix , Indicates the first The learnable transformation matrix of the layer, Represents a node The neighboring nodes, Indicates the first The node feature matrix of the layer.

7. The lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution as described in claim 4, characterized in that, The nonlinear excitation uses the rectified linear unit ReLU as the feature map. The nonlinear activation function is used to transform the feature map Each value in the equation undergoes a non-linear transformation, and the rectified linear unit ReLU is defined as follows: in, This indicates finding the maximum value. It is an input value.

8. The lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution as described in claim 4, characterized in that, The probability value conversion uses The function transforms the predicted class scores into a probability distribution. The function is defined as follows: in, Is the pixel belonging to the first The probability of a class , It is the pixel in the final output of the segmentation network at the 1st position. , No. Predicted scores for each class Represents the number of categories.

9. The lightweight retinal vessel segmentation method based on graph convolutional networks and partial convolution as described in claim 1, characterized in that, The specific optimization process in step S26 is as follows: in, Represents the number of steps updated. For the network parameters that need to be updated, corresponding to the objective optimization function in step S25... , For parameters loss function, For the target loss function right The gradient obtained by differentiation, The first-order moment attenuation coefficient, The second-order moment attenuation coefficient, gradients The first and second moments, for bias correction, for bias correction, This represents the learning rate, used to control the stride length; the initial value is set to 5e. -4 The cosine annealing algorithm is used to gradually decay the learning rate to 1e. -5 , It is a small positive number to prevent the denominator from being zero.

Images