Retinal vessel image segmentation method based on multi-scale dilated convolution residual network

Abstract

In order to solve the problems of limited labeled data, differences between blood vessels and interference of lesion area, the present application discloses a retinal blood vessel image segmentation method based on a multi-scale dilated convolution residual network, which is still a challenge problem for accurately segmenting retinal blood vessels, especially fine blood vessels, on a retinal fundus image. The present application designs a multi-scale residual input and output module to make up for the loss of part of the blood vessel structure information due to down-sampling, combines the advantages of dilated convolution and DropBlock to relieve network overfitting and reduce the influence of the lesion area on blood vessel feature extraction, and further introduces a multi-scale mean pooling module to obtain high-level features and retain context information. Finally, by improving the way of jump connection, the dilated convolution is effectively used to improve the information transmission capacity of the jump connection. Compared with other algorithms, the present application can more accurately segment the fine blood vessels in the retinal image under complex conditions and has better robustness.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and in particular to a method for retinal vessel image segmentation based on a multi-scale dilated convolutional residual network. Background Technology

[0002] Retinal diseases such as diabetic retinopathy, glaucoma, and age-related macular degeneration are leading causes of blindness in the elderly. Clinically, observing changes in retinal vessels in fundus images helps in diagnosing these diseases and making subsequent treatment decisions. This process requires highly skilled physicians to manually label retinal vessels, which is time-consuming, labor-intensive, and the labeling results are susceptible to subjective factors. Automated computer-aided segmentation of retinal vessels is of great significance in the diagnosis of these diseases. Retinal vessel segmentation has always been a challenging task in the field of medical image segmentation. Current methods still have room for improvement in the accuracy of retinal vessel segmentation, especially for small retinal vessels.

[0003] Currently, retinal vessel segmentation methods are broadly classified into two categories: supervised learning methods and unsupervised methods. Unsupervised methods attempt to achieve segmentation using the fixed structure of retinal vessels. These methods mainly include mathematical morphology methods, matched filtering methods, multi-scale methods, and region growing methods. Their advantage lies in the absence of manually labeled datasets. However, these methods suffer from two main drawbacks: First, due to the lack of global contextual information, their segmentation accuracy is relatively low. Second, these methods rely on hand-designed feature extractors, while retinal images contain complex and diverse backgrounds, such as infectious lesions, which often lead to false positives or false negatives in the segmentation results. Compared to supervised methods, unsupervised methods exhibit poor robustness and limited performance.

[0004] In recent years, supervised deep learning methods have become the mainstream approach for retinal vessel segmentation. Among them, the U-Net deep learning network, given its outstanding performance in medical image segmentation tasks, has been widely applied to retinal vessel segmentation. Compared to traditional unsupervised methods, U-Net-based methods can automatically learn complex features, improving the accuracy of retinal vessel segmentation. However, using the original U-Net network for retinal vessel segmentation presents the following problems: While the network uses a skip connection mechanism to fuse low-level and high-level features to address the spatial information loss caused by downsampling in deep convolutional neural networks, semantic feature differences make it difficult for the decoder to recover some spatial information from the shallow stages of the encoder. Furthermore, the U-Net network still has limitations when dealing with small and irregular retinal vessel structures and limited labeled data. In recent years, researchers have further improved the retinal vessel segmentation performance of U-Net. For example, the SD-UNet network model alleviates overfitting by introducing a DropBlock structure into the U-Net architecture. SA-UNet introduces batch normalization (BN) layers into the convolutional blocks of SD-UNet, utilizing spatial attention mechanisms to improve the network's feature extraction capabilities. DRNet improves upon U-Net's skip connections using a dense connection method. By combining residual and DropBlock structures, it reduces the semantic gap between the encoder and decoder, increases network depth, and alleviates overfitting. While these methods further improve retinal vessel segmentation based on U-Net, they primarily address network overfitting, neglecting the structural characteristics of retinal vessels, especially small ones. For the curved structure of retinal vessels, CS... 2 -Net utilizes 1×3 and 3×1 convolutions to extract vascular morphological features in two directions, highlighting regions of interest through channel and spatial attention mechanisms. To address the discontinuous segmentation problem in vascular segmentation results, FR-UNet expands in the horizontal and vertical directions through a multi-resolution convolutional interaction mechanism while maintaining the full image resolution, and then uses a dual-threshold iterative algorithm to extract fine vessel pixels to improve vessel connectivity. To address the spatial information loss caused by continuous convolution and pooling operations, CE-Net constructs a context extraction module using dense dilated convolutions and residual multi-kernel pooling to obtain more contextual information. These methods significantly improve the retinal vessel segmentation capability of the model by adding extra modules between the U-Net encoder and decoder to obtain more high-level semantic features. However, these methods neglect the importance of low-level spatial information for the segmentation of small retinal vessels. Although the above deep learning methods have designed meaningful models to handle the retinal vessel segmentation problem from different perspectives, their segmentation accuracy for small vessels, especially under complex conditions such as interference from lesion areas and low contrast, still needs improvement. Summary of the Invention

[0005] The purpose of this invention is to automatically and accurately segment blood vessels in retinal images using deep learning methods. A multi-scale dilated convolutional residual network model for retinal vessel segmentation is proposed to solve the problem of inaccurate segmentation of retinal vessels in fundus images, especially under complex conditions such as interference from lesion areas and low contrast.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: a retinal vessel image segmentation method based on a multi-scale dilated convolutional residual network, the method comprising the following steps:

[0007] Step 1: Preprocess the retinal image;

[0008] Step 2: The preprocessed retinal images are augmented using random horizontal flipping, vertical flipping, random rotation with an angle range of [0, 360], and random cropping to increase the number of training images and obtain the model training set;

[0009] Step 3: Improve the existing U-Net network by incorporating the characteristics of retinal images in three aspects: encoder, decoder, skip connections, and loss function. Add a multi-scale residual input (MRI) module, a multi-scale residual output (MRO) module, and a multi-scale mean pooling module. In each layer of the encoder, the MRI module processes the input image at different scales. The output of each layer's MRI module is passed to the combined dilated convolution (DC) module, which is composed of dilated convolution with different dilation rates, DropBlock, batch normalization (BN), and ReLU activation function, to obtain the output features of each layer of the encoder. Each layer of the encoder is connected to the DC module of the corresponding decoder layer through the skip connection DRes Path module. Except for the first layer of the decoder, the multi-scale residual output (MRO) module of each layer of the decoder receives the features output by the corresponding decoder layer's DC module and generates output feature maps at different scales. The output feature maps at different scales are weighted and fused to obtain the retinal segmentation result image.

[0010] Step 4: Use the binary cross-entropy loss function and the Dice loss function to form a hybrid loss function. Use the training dataset obtained in Step 2 to train the network model designed in Step 3 until the hybrid loss function converges to the optimal value.

[0011] Preferably, the preprocessing operation in step 1 is as follows: after grayscale processing, an adaptive histogram equalization method with limited contrast and a gamma correction method are used to enhance image contrast and highlight retinal vascular structures.

[0012] Preferably, step 3 introduces multi-scale information in the encoder section, and passes the input images downsampled at different scales to the multi-scale residual input (MRI) module to extract features at different scales.

[0013] The multi-scale residual input (MRI) module consists of two convolutional layers. After each convolutional layer, DropBlock, batch normalization, and linear rectified activation functions are applied. The first layer uses dilated convolution to increase the number of image channels to obtain higher-level representation features. The second layer uses 1×1 convolution to reduce the number of channels and compress information to reduce network complexity and improve computational efficiency, avoiding the depth explosion problem. The sequence of these two consecutive convolutional layers is executed three times on the input image, each time using a different dilation rate. Then, these three output feature maps are jointly stitched together, and the jointly stitched feature map is pixel-wise added to the original input image through residual connections to fully utilize multi-scale features and enhance feature transfer capability, resulting in the MRI module output feature map. To ensure that the output features of the MRI module match the output features of the stitched dilated convolution module, the number of channels in the MRI module increases with the network depth.

[0014] Preferably, in step 3, in addition to the first layer of the decoder, a multi-scale residual output (MRO) module is added after the combined dilated convolution module of each layer of the decoder to extract features at different scales.

[0015] The Multi-Scale Residual Output (MRO) module adds an upsampling method (common upsampling methods include deconvolution (also known as transposed convolution) or bilinear interpolation, which can be selected as needed to meet the requirements of a specific task) to the MRI module to ensure that the output feature map of each MRO is the same size as the output feature map of the decoder 3DC block. The outputs of the three MRO modules and the 3DC module in the decoder are all 1-channel feature maps, which are weighted and concatenated to form a 4-channel feature map, where the weighting coefficients w1 to w4 take values ​​from 0 to 1. Finally, this feature map is processed by applying a 1×1 convolution and a sigmoid activation function to produce a probability value image with values ​​ranging from 0 to 1.

[0016] Preferably, in step 3, the network introduces a combined dilated convolution (DC) module consisting of dilated convolution with different dilation rates, DropBlock, batch normalization (BN), and ReLU activation function. This combination structure with dilation rates of 1, 2, and 3 is named 1DC module, 2DC module, and 3DC module. These DC modules can accelerate network training convergence and effectively alleviate the overfitting problem of convolutional networks.

[0017] In step 3, the improved U-Net model gradually increases the expansion rate of the DC module in the left encoder as the depth increases, while gradually decreasing the expansion rate of the DC module in the right decoder as the depth decreases.

[0018] As a preferred option, step 3 replaces the skip connections in U-Net with the DRes Path module, and performs feature transformation in the skip connections, thereby better fusing features and preserving important details.

[0019] The DRes Path structure is as follows: Unlike U-net, which directly passes the encoder's corresponding layer features to the decoder's corresponding layer, the improved skip connection first fuses the encoder's corresponding layer's output features with the result of this feature dilation, and then passes the fused features to the decoder's corresponding layer. Figure 1 In this design, because the number of convolutional layers between the encoder and decoder connected by different skip connections varies, the difference between the shallow features obtained by the encoder on the left and the deep features obtained by the decoder on the right is relatively small compared to higher-level skip connections. To address this variation, the number of dilated convolutions in the four skip connections is gradually reduced as the network depth increases. The usage of these convolutions and their dilation rates is shown in Table 3. This design allows the network to better integrate semantic features of different depths. Furthermore, since residual connections help accelerate network convergence and improve network performance, this invention introduces residual connections into the DRes Path.

[0020] Preferably, step 3 adds a multi-scale average pooling (MAP) module between the convolutional layers of each encoder and decoder in U-Net. The MAP module addresses the issue of varying object sizes during segmentation, and the MAP module outputs Z∈R. 4c×h×w Where c is the number of image channels, h is the image height, w is the image width, and Z is calculated as follows:

[0021]

[0022] Z * =[Z2,Z3,Z5,Z7,S],

[0023]

[0024] Where S∈R 4c×h×w This indicates the characteristics of the input MAP module, where U represents the upsampling operation, and P represents the input MAP module. i This is an average pooling operation, where i represents the size of the pooling kernel, and i∈{2,3,5,7}, K1∈R. c×1×1 and K2∈R 8c×1×1 [ ] represents the convolution kernel, and [ ] represents the convolution operation. Represents standard convolution;

[0025] The pyramid-structured MAP module encodes the input feature map S to capture global contextual information. This pyramid structure uses four receptive fields of different sizes for pooling operations. To balance the increase in parameters, after each pyramid pooling layer, the feature map undergoes dimensionality reduction via a 1×1 convolution, reducing the number of channels to 1 / N of the original number, where N is the number of pyramid layers. Then, bilinear interpolation is used to upsample the 1×1 convolution result. Finally, the input feature map is concatenated with all the upsampled feature maps to obtain Z. * However, Z * There are semantic incompatibility issues between different feature maps, and direct fusion may lead to inaccurate segmentation. In order to effectively fuse features, the MAP module introduces another 1×1 convolutional layer to Z. * This convolutional layer reduces feature dimensionality while achieving effective feature fusion.

[0026] Preferably, in step 3, the U-Net network uses the binary cross-entropy loss function L. bce and Dice loss function L dice Composed of a hybrid loss function, the binary cross-entropy loss function L bce and Dice loss function L dice The calculations are shown in the following formulas:

[0027]

[0028]

[0029] Where P(i) is the model prediction value, P(i)∈(0,1), and its value reflects the probability that pixel i is predicted as a blood vessel pixel: the larger the value, the greater the probability; G(i) is the label, which takes the value 0 or 1; n is the number of pixels in the image; the total loss function is expressed as:

[0030] L total =L bce +L dice .

[0031] Preferably, during model training in step 3, the Adam optimizer is used with a learning rate of 0.000001 to 0.001, 150 to 250 training iterations, a batch size of 4 to 8, a DropBlock size of 4 to 7 for each dataset, and the output probability of each neuron is kept at 0.1 to 0.9.

[0032] This invention uses expert-segmented results provided by the dataset as the label (Ground Truth) image, and introduces accuracy (Acc), sensitivity (Se), specificity (Sp), F1 score, and area under the curve (AUC) as evaluation metrics for the segmentation results. The calculation methods for Acc, Se, Sp, and F1 are as follows:

[0033]

[0034]

[0035]

[0036]

[0037] TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively. During calculation, the values ​​of TP, TN, FP, and FN are obtained by comparing the retinal vessel segmentation results of the test method with the corresponding pixels in the labeled image (Ground Truth). AUC is defined as the area under the Receiver Operating Characteristic Curve (ROC) and the coordinate axes. The values ​​of Acc, Se, Sp, and F1 range from 0 to 1, and the AUC ranges from 0.5 to 1. The larger the values ​​of these five indicators, the better the model's classification effect, i.e., the better the retinal vessel segmentation effect.

[0038] The features of this invention are as follows: This invention preprocesses the original retinal vessels using methods such as adaptive histogram equalization and gamma correction; it increases the number of training images using data augmentation methods such as horizontal flipping, vertical flipping, random rotation, and random cropping to obtain a training set. The U-Net model is improved in three aspects—encoder, decoder, skip connections, and loss function—incorporating the characteristics of retinal images, while also adding MRI, MRO, and MAP modules. Finally, the improved network is trained using the training set to obtain a model capable of accurately segmenting retinal vessels.

[0039] Compared with existing technologies, this invention has the following advantages: By designing multi-scale residual input and output modules, this invention compensates for the loss of some vascular structure information due to downsampling. By combining the advantages of dilated convolution and DropBlock, it alleviates network overfitting and reduces the impact of lesion areas on vascular feature extraction. In addition, a multi-scale mean pooling module is introduced to obtain high-level features and retain contextual information. Finally, by improving the skip connection method, dilated convolution is effectively used to improve the information transmission capability of skip connections. Compared with other algorithms, this invention can more accurately segment small blood vessels in retinal images under complex conditions and has better robustness.

[0040] This invention achieves better retinal vessel segmentation, especially in accurately segmenting small blood vessel structures in retinal images; furthermore, this invention can more accurately segment blood vessel structures in lesion areas of retinal vascular images. Attached Figure Description

[0041] Figure 1 This is a diagram of the network structure designed in this invention;

[0042] Figure 2 (a) and (b) are structural diagrams of the multi-scale input MRI module and the multi-scale output MRO module of the present invention, respectively.

[0043] Figure 3 This is a diagram of the internal structure of the DRes Path1 module of the present invention;

[0044] Figure 4 This is the internal structure of the multi-scale mean pooling module of the present invention;

[0045] Figure 5 Preprocess the image;

[0046] Figure 6 This is a visual comparison of the retinal vessel segmentation results of the present invention and the U-Net network on the DRIVE, CHASE_DB1, STARE and HRF datasets;

[0047] Figure 7 This is a local comparison between the present invention and the U-Net network in fine blood vessel segmentation;

[0048] Figure 8 This is a visual comparison of the segmentation results of the present invention and various retinal vessel segmentation methods on the DRIVE, CHASE_DB1, STARE, and HRF datasets;

[0049] Figure 9 This is a comparison between the retinal vessel segmentation results of this invention and Ground Truth;

[0050] Figure 10 It is a comparison of local areas of the retinal vessel segmentation results. Detailed Implementation

[0051] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, but the present invention is not limited to the following technical solutions. This embodiment uses the following four publicly available retinal image datasets: DRIVE (https: / / drive.grand-challenge.org / ), STARE (http: / / cecas.clemson.edu / ~ahoover / stare / ), CHASEDB1 (https: / / blogs.kingston.ac.uk / retinal / chasedb1 / ), and HRF (https: / / www5.cs.fau.de / research / data / fundus-images / ).

[0052] Example 1

[0053] A method for retinal vessel image segmentation based on a multi-scale dilated convolutional residual network, the specific steps of which are as follows:

[0054] Step 1: Preprocess the retinal image, which includes using an adaptive histogram equalization method with limited contrast and a gamma correction method to enhance image contrast and highlight the vascular structure of the retina;

[0055] Step 2: Data augmentation is performed using methods such as random image horizontal flipping with a probability of 0.5, random image vertical flipping with a probability of 0.5, random image rotation within the angle range of [0, 360], and random image cropping to obtain an image training set. The number of image pairs in the training set is four times the number of image pairs before data augmentation.

[0056] Step 3: Improve the U-Net network in five aspects to enhance its ability to accurately segment retinal vessels;

[0057] Step 4: Use the binary cross-entropy loss function and the Dice loss function to form a hybrid loss function. Use the training dataset generated in Step 2 to train the network model designed in Step 3 until the hybrid loss function converges to the optimal value. Use the trained model to perform blood vessel segmentation on retinal images that were not used in the training to obtain segmented images of retinal vessels.

[0058] This invention employs supervised deep learning and consists of two phases: training and testing. The training phase involves the following steps: First, preprocessing and image augmentation are performed on four publicly available retinal image datasets: DRIVE, STARE, CHASEDB1, and HRF, resulting in four datasets. These datasets are then divided into training and testing sets. Next, the network model designed in this invention is trained using the training set, while the testing set is not used in the training. Finally, in the testing phase, the trained model is used to segment blood vessels in the retinal images of the testing set. Specifically:

[0059] 1. Data Preprocessing

[0060] This embodiment selects four publicly available retinal image datasets: DRIVE, STARE, CHASEDB1, and HRF. The DRIVE dataset contains 40 color images of the fundus retina (33 from non-diabetic individuals and 7 from patients with mild diabetic retinopathy). The STARE dataset contains 20 color images of the fundus retina (10 from diseased retina and 10 from normal retina). The CHASEDB1 dataset contains 28 color images of the fundus retina, taken from the binocular retina of 14 school children. The HRF dataset contains 45 color images of the fundus retina (15 from healthy individuals, 15 from patients with diabetic retinopathy, and 15 from patients with glaucoma).

[0061] Because the color fundus images in the retinal dataset exhibit significant tonal variations and contrast differences, each image is first processed to grayscale to reduce these interfering factors. It was observed that the contrast between blood vessels and the background in the retinal images was low, resulting in insufficient clarity of vascular details. To address this issue, this invention employs a Contrast Limited Adaptive Histogram Equalization (CLAHE) method to enhance the contrast between blood vessels and the background. The CLAHE method can equalize the histogram in local regions, significantly improving the contrast of retinal images. Then, gamma correction is applied to the CLAHE processing results. Gamma correction effectively highlights the darker vascular structures in the retinal images, further enhancing the contrast of retinal vessels. Figure 5 The results of the image preprocessing steps for the DRIVE dataset are shown ((a) the original image from the DRIVE dataset, (b) the image after grayscale processing, (c) the image after CLAHE processing, and (d) the image after gamma correction).

[0062] 2. Data Augmentation

[0063] Since the four datasets contain a relatively small number of images (133 images in total), training the network using only these images would lead to overfitting. Therefore, this invention employs data augmentation methods. These include: random horizontal flipping with a probability of 0.5, random vertical flipping with a probability of 0.5, random rotation within the range [0, 360], and random cropping. These methods increase the number of training images to four times the original number, alleviating the network overfitting problem. Table 1 shows the number of images in each dataset, the training / test set split, the resolution before and after cropping, and the number of training images after data augmentation.

[0064] Table 1 details the four datasets.

[0065]

[0066] 3. Improve the U-Net network

[0067] The U-Net network model has been improved to enhance its ability to accurately segment retinal vessels; the improved network structure is as follows: Figure 1 As shown. Specific improvements include: refining the original U-Net in three aspects—encoder, decoder, skip connections, and loss function—to suit the characteristics of retinal images; and adding a multi-scale input module, a multi-scale output module, and a multi-scale mean pooling module. The five improvements are described in detail below:

[0068] (1) Introduce MRI and MRO modules

[0069] This invention introduces improved MRI and MRO modules based on the U-Net network architecture, the structures of which are as follows: Figure 2 As shown in (a) and (b), the MRI module consists of two convolutional layers. Each convolutional layer is followed by DropBlock, batch normalization, and a linear rectified activation function. The first layer uses dilated convolution to increase the number of image channels to obtain higher-level representation features. The second layer uses a 1×1 convolutional operation to reduce the number of channels and compress information, thereby reducing network complexity, improving computational efficiency, and avoiding depth explosion. These two consecutive convolutional layers are executed three times on the input image, each time using a different dilation rate. The advantage is that by using different receptive fields, it effectively reduces interference from lesion areas in vessel segmentation and obtains more information about small blood vessels in lesion areas. Subsequently, these three output feature maps are jointly stitched together, and the jointly stitched feature map is pixel-wise added to the original input image through residual connections to fully utilize multi-scale features and enhance feature transfer capability, resulting in the MRI module output feature map. To ensure that the output features of the MRI module match the output features of the stitched dilated convolution (DC) module (which combines DropBlock structures), the number of channels in the MRI module increases with network depth. Table 2 provides... Figure 1 Detailed information on the number of channels in each stage of the four MRI modules. The internal structure of the MRI modules is as follows: Figure 2 As shown in (a).

[0070] Table 2 Number of channels per MRI module

[0071]

[0072] Unlike MRI structures, MRO incorporates an upsampling method to ensure that each MRO output feature map is the same size as the decoder 3DC block output feature map. Figure 1In this model, the outputs of the three MRO modules and the 3DC module in the decoder are all 1-channel feature maps. These are weighted and concatenated to form a 4-channel feature map, where the weighting coefficients w1, w2, w3, and w4 are 1, 0.6, 0.4, and 0.2, respectively. Finally, this feature map is processed by applying a 1×1 convolution and a sigmoid activation function to produce a probability value image with values ​​ranging from 0 to 1. This weighted fusion idea stems from the fact that multi-scale feature maps contain different levels of semantic and detail information. Small-scale feature maps provide higher-level semantic information, while large-scale feature maps retain more detail information. By weighted fusion of these multi-scale outputs, the contributions of features at different scales can be balanced. Weighting assigns higher weights to large-scale feature maps, enhancing the model's focus on detail information and reducing segmentation errors for detailed structures such as small blood vessels.

[0073] (2) Employing structured DropOut convolutional blocks and effectively using dilated convolutions

[0074] Due to the limited number of samples in currently available retinal datasets, overfitting still occurs during training despite data augmentation techniques. To address this issue, this invention employs a structured packet dropping method, namely the DropBlock structure. The DropBlock structure prevents overfitting by discarding adjacent contiguous regions in the feature map. Although this method removes some semantic information, the remaining units are forced to learn features to classify the output image, thus avoiding network overfitting. Unlike the U-Net network encoder and decoder which uses 3×3 convolutions to extract image features, this invention introduces 3×3 convolution operations with different dilation rates, followed by DropBlock blocks, batch normalization (BN), and a ReLU activation function. This invention names this combined structure 1DC, 2DC, and 3DC modules based on their dilation rates. These DC modules accelerate network training convergence and effectively alleviate the overfitting problem in convolutional networks.

[0075] To simultaneously meet the requirements of a large receptive field and high spatial resolution, this invention uses dilated convolution instead of ordinary convolution. In blood vessel segmentation tasks, dilated convolution with different dilation rates can introduce richer local contextual information, helping to mitigate semantic misunderstandings that may be caused by 3×3 convolutions in the U-Net network. The model proposed in this invention gradually increases the dilation rate of the DC module in the left encoder as depth increases, while gradually decreasing the dilation rate of the DC module in the right decoder as depth decreases. It should be noted that while dilated convolution with different dilation rates in the encoder stage can acquire spatial information at different scales in the initial stage, the sparse connectivity of the dilation kernel means that excessively increasing the dilation rate leads to a decrease in spatial consistency between adjacent information units. Simultaneously, at higher levels, higher-level units can only acquire partial information from non-overlapping units, resulting in the loss of local structural information due to difficulty in extraction. This local structural information is crucial for detecting smaller objects (e.g., tiny blood vessels). Therefore, this invention restores spatial consistency by reducing the dilation rate in the decoder stage. This effective use of the dilated convolution sequence is also applicable to subsequent skip connection parts. By applying dilated convolution operations appropriately and using different dilation rates according to changes in network depth, this invention can balance the requirements of a large receptive field and high spatial resolution in blood vessel segmentation tasks, thereby improving the ability to accurately detect local structures such as small blood vessels.

[0076] (3) Introduce the DRes Path module for skip connections

[0077] The U-Net network preserves image details by propagating spatial information lost in the encoder to the decoder through skip connections between the encoder and decoder. However, there are significant semantic differences between the shallow features obtained by the encoder and the deep features obtained by the corresponding decoder layers. This difference reduces the effectiveness of feature fusion. To reduce the feature differences between the encoder and decoder, this invention adds convolutional layers to the skip connections. Specifically, this invention replaces the skip connections in U-Net with DRes Path modules. This design performs feature transformation within the skip connections, thereby better fusing features and preserving important details. Figure 1 The first DRes Path structure from top to bottom is as follows: Figure 3 As shown, 3×3 dilated convolutions with dilation rates of 1, 2, and 3 are used to incorporate skip connections to capture points of interest in images at different scales. Figure 1 As shown, due to the different number of convolutional layers between the encoder and decoder connected by different skip connections, the difference between the shallow features obtained by the left encoder and the deep features obtained by the right decoder in lower-level skip connections is relatively small compared to higher-level skip connections. To address this variation, the number of dilated convolutions in the four skip connections is gradually reduced as the network depth increases. The usage of their number and dilation rate is shown in Table 3, where DResPath1 to 4 correspond to... Figure 1The network consists of four DRes Path structures arranged from top to bottom. This design allows the network to better integrate semantic features of different depths. Furthermore, since residual connections help accelerate network convergence and improve network performance, this invention introduces residual connections into the DRes Path.

[0078] Table 3 Internal structure of DRes Path

[0079]

[0080] (4) Multi-scale mean pooling (MAP) module

[0081] To effectively aggregate and process more contextual information, this invention uses a MAP module to address the problem of varying object size during segmentation. This invention relies on multiple effective receptive fields to detect objects of different sizes; the receptive field size roughly indicates how much contextual information the model uses. Compared to U-Net, which only uses 3×3 convolutions to extract image features, the multi-scale approach can preserve multi-scale global information. The MAP module outputs Z∈R. 4c×h×w The calculation is as follows:

[0082]

[0083] Z * =[Z2,Z3,Z5,Z7,S],

[0084]

[0085] Where S∈R 4c×h×w This indicates the characteristics of the input MAP module, where U represents the upsampling operation, and P represents the input MAP module. i This is an average pooling operation, where i represents the size of the pooling kernel, and i∈{2,3,5,7}. K1∈R c×1×1 and K2∈R 8c×1×1 [ ] represents the convolution kernel, and [ ] represents the convolution operation. This represents standard convolution.

[0086] like Figure 4 As shown, the pyramid-structured MAP module encodes the input feature map S to capture global contextual information. This pyramid structure uses four different receptive field sizes (2×2, 3×3, 5×5, and 7×7) for pooling. To balance the increase in parameters, the feature map after each pyramid pooling layer undergoes dimensionality reduction via a 1×1 convolution, reducing the number of channels to 1 / N of the original (N is the number of pyramid layers, which is 4 in this invention). Then, the 1×1 convolution result is upsampled using bilinear interpolation. Finally, the input feature map is concatenated with all upsampled feature maps to obtain Z. * However, Z *Semantic incompatibility exists between different feature maps, and direct fusion may lead to inaccurate segmentation. To effectively fuse features, the MAP module introduces another 1×1 convolutional layer on Z. * The convolutional layer reduces feature dimensionality while achieving effective feature fusion, enhancing the high-level feature capture capability of MAP and improving the processing capability of the MAP module in the task of segmenting small retinal vessels, thereby improving the accuracy and reliability of the segmentation results.

[0087] (5) Loss Function

[0088] In retinal image segmentation tasks, the choice of loss function is crucial for accurately segmenting important structures in the retina (such as blood vessels and lesions). In retinal images, background pixels typically constitute the majority, while target structures of interest (such as blood vessels) are only a minority, leading to class imbalance. Furthermore, the boundaries of some structures are not clearly defined, such as the transition region between blood vessels and surrounding tissue. This can cause traditional loss functions to produce significant errors. Based on these characteristics of retinal images, this invention employs a hybrid loss function composed of binary cross-entropy loss and Dice loss. The calculations of the binary cross-entropy loss function and the Dice loss function are shown in the following equations:

[0089]

[0090]

[0091] Where P(i) is the model prediction value, P(i)∈(0,1), and its value reflects the probability that pixel i is predicted as a blood vessel pixel: the larger the value, the greater the probability. G(i) is the label, taking the value 0 or 1. n is the number of pixels in the image. The total loss function is expressed as:

[0092] L total =L bce +L dice .

[0093] 5. Network training and experimental testing

[0094] The Adam optimizer was used during training. The learning rate was set to 0.001, and the network was trained for 150 iterations. The batch size was 4. The DropBlock size for each dataset was set to 7, and the output probability of each neuron was kept at 0.9. Table 4 provides the batch size and drop rate parameter settings for the four datasets.

[0095] Table 4. Batch size and Dropout rate for the four datasets: DRIVE, STARE, CHASE_DB1, and HRF.

[0096]

[0097] (1) Implementation Environment

[0098] The hardware environment in this embodiment is as follows: Dell Intel Xeon Gold 6226R processor and Dell RTX A6000 graphics card; the system software environment is Windows 10 (64-bit) operating system; the development and testing software environment is PyCharm Community Edition 2022.2.3 x64. The programming language used is Python, and the software environment is based on the CUDA 11.0, CUDN 11.0, and PyTorch 1.12.1 deep learning framework.

[0099] (2) Evaluation indicators

[0100] To quantitatively evaluate the retinal vessel segmentation effect of the proposed model, this embodiment uses the expert manual segmentation results provided by the dataset as the ground truth image, and introduces accuracy (Acc), sensitivity (Se), specificity (Sp), F1 score, and area under the curve (AUC) as evaluation indicators for the segmentation results. The calculation methods for Acc, Se, Sp, and F1 are as follows:

[0101]

[0102]

[0103]

[0104]

[0105] TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively. During calculation, the values ​​of TP, TN, FP, and FN are obtained by comparing the retinal vessel segmentation results of the test method with the corresponding pixels in the labeled image (Ground Truth). AUC is defined as the area under the Receiver Operating Characteristic Curve (ROC) and the coordinate axes. Acc, Se, Sp, and F1 values ​​range from 0 to 1, and AUC values ​​range from 0.5 to 1. Higher values ​​for these five indicators indicate better model classification performance, i.e., better retinal vessel segmentation.

[0106] (3) Model training and testing

[0107] The network model designed in this invention is trained using the training dataset obtained after data preprocessing and data augmentation in this embodiment until the loss function converges to the optimal value, thus obtaining a trained model. The retinal images of the test set that were not involved in the training are input into the trained model to obtain retinal vessel segmentation images. The retinal vessel segmentation results of this invention are compared with other advanced methods in terms of visualization, quantification, and complex environment.

[0108] 1) Visual comparison of this invention with other algorithms

[0109] Figure 6 The visualizations show the retinal vessel segmentation results of this invention and the U-Net network on the aforementioned datasets ((a) original image, (b) ground truth, (c) U-Net network segmentation result, (d) segmentation result of this invention). On the DRIVE and STARE datasets, the U-Net network exhibits the following problems in small vessel segmentation: it fails to successfully segment small vessels and the segmentation results contain broken small vessels. On the CHASE-DB1 and HRF datasets, the U-Net network cannot effectively handle interference from lesion areas, mistakenly segmenting background pixels as white vessel pixels. In contrast, this invention can segment small vessels more accurately while maintaining the integrity of the entire vessel segmentation. Furthermore, this invention has stronger anti-interference capabilities when dealing with lesion areas. Figure 7 The results of the comparative experiments on the visualization of fine blood vessel segmentation in the DRIVE dataset using the present invention and U-Net are shown. Figure 7 (c) and Figure 7 As shown by the yellow arrow in the green box in (d), this invention can segment small blood vessels that exist in the Ground Truth but were not segmented by the U-Net network. Furthermore, the U-Net network's segmentation results for small blood vessels show breaks (e.g., ...). Figure 7 (c) As shown by the yellow arrow in the red box), the present invention obtained a more accurate segmentation result (e.g. Figure 7 (d) As indicated by the yellow arrow in the red box.

[0110] Under the same experimental conditions, Figure 8 This invention (MDR-Net) is shown in comparison with SD-UNet, SA-UNet, and CS. 2 Visual comparison results of four methods, including SD-Unet, FR-UNet, etc., on four datasets ((a) original image, (b) ground truth, (c) SD-Unet network segmentation result, (d) SA-UNet network segmentation result, (e) CS2-Net network segmentation result, (f) FR-UNet network segmentation result, (g) segmentation result of the present invention). Figure 8The images show localized images of tiny blood vessels and their magnified versions, marked with green and red borders. In the DRIVE dataset experiments, the four methods compared showed instances of missed detections, incomplete segmentation, or breakage in the segmentation of small blood vessels (indicated by yellow arrows within the red and green boxes), while this invention segmented the small blood vessels more accurately. In the CHASE_DB1 dataset experiments, the four methods compared failed to completely segment the ends of the small blood vessels within the green box, while this invention segmented the small blood vessels completely. In the STARE dataset experiments, CS... 2 While -Net can segment the small blood vessels within the green box, compared to the label (Ground Truth), CS... 2 The Net method segmented small blood vessels with excessively large diameters, while this invention can segment these small blood vessels more accurately. Similarly, in the HRF dataset experiments, this invention can segment small blood vessels more accurately than the comparison method. These experimental results demonstrate that the present invention provides more accurate small blood vessel segmentation results compared to the comparison method.

[0111] 2) Quantitative comparison experiment between this invention and other algorithms

[0112] To quantitatively evaluate the retinal vessel segmentation performance of this invention, quantitative comparisons were performed with widely accepted retinal vessel segmentation methods on four public datasets. In this implementation example, we reimplemented U-Net, SD-UNet, SA-UNet, and CS. 2 This invention compares five methods, including -Net and FR-UNet, with those methods. For other methods, since no code is provided, comparative data from relevant literature is used as a reference. Quantitative evaluation metrics include five key metrics: sensitivity (Se), specificity (Sp), accuracy (Acc), F1 score, and area under the curve (AUC). The quantitative comparison experimental results are shown in Table 5, where the best-performing data are marked in bold, and the second-best data are marked with an underline. "-" indicates that the corresponding experimental results were not provided in the paper.

[0113] The comparative experimental results on the DRIVE dataset are shown in Table 5. Compared with other methods in Table 5, the present invention performs best in Se, Acc, F1, and AUC metrics. Although SD-UNet achieves the best results in Sp, which measures the accuracy of background segmentation, it lags behind the present invention in metrics such as Se, Acc, F1, and AUC. In particular, in Se, a key metric for evaluating vessel segmentation performance, SD-UNet is 8.81% lower than the present invention.

[0114] Table 5. Comparison of experimental results between the present invention and other methods on the DRIVE dataset.

[0115]

[0116] Table 6 presents the quantitative comparative experimental results of the present invention and the comparative method on the STARE dataset. As shown in Table 6, the method of the present invention exhibits the best overall performance, with the present invention achieving a 4.11% higher Se score than the second-best FR-Unet. Compared to the method proposed by Guo et al., the F1 score of the present invention is slightly lower, with a difference of 2.09%, but it increases Se, Sp, Acc, and AUC by 5.52%, 0.25%, 0.89%, and 0.28%, respectively. It is worth noting that the reason why the present invention's method is slightly lower than the best method in Sp, Acc, and F1 scores is due to the presence of unlabeled microvessels (e.g., small blood vessels) in the STARE dataset's ground truth images. Figure 9 As shown in the diagram, this invention is able to successfully segment these tiny blood vessels. These unlabeled tiny blood vessels resulted in a slightly lower score for the corresponding indicators of this invention.

[0117] Table 6. Comparison of experimental results between the present invention and other methods on the STARE dataset.

[0118]

[0119] The quantitative experimental results of this invention and the comparative methods on the CHASE-DB1 and HRF datasets are shown in Tables 7 and 8. In the CHASE-DB1 dataset, this invention achieves optimal performance in all five metrics except for the Sp metric. On the HRF dataset, MDR-Net performs best in all five metrics. In the CHASE-DB1 dataset, although the Sp metric of this invention is 0.64% lower than that of FR-Unet, it outperforms FR-Unet in other metrics such as Se, Acc, F1, and AUC, especially in the Se metric, which evaluates the correct segmentation of vessel pixels, where it is 5.66% higher. Furthermore, the Se metric of this method is 4.19% higher than that of the suboptimal method Tomar et al. Meanwhile, on the HRF dataset, the Se and F1 metrics of this invention are 2.35% and 3.38% higher than those of Li et al. and SD-Unet, respectively. These quantitative experimental results fully demonstrate the excellent performance of this invention in retinal vessel segmentation.

[0120] Table 7 Comparative experimental results of the present invention and the comparative method on the CHASE-DB1 dataset.

[0121]

[0122]

[0123] Table 8 Comparative experimental results of the present invention and the comparative method on the HRF dataset.

[0124]

[0125]

[0126] 3) Performance of the method of the present invention in areas where retinal vessel segmentation is difficult.

[0127] The complex tortuous structure of retinal vessels, low-contrast retinal images, and lesions affecting retinal vessels make retinal vessel segmentation extremely challenging. Figure 10 The image shows the retinal vessel segmentation results of this invention and five other methods under complex conditions ((a) original image, (b) Ground Truth, (c) U-Net network segmentation results, (d) SD-Unet network segmentation results, (e) SA-UNet network segmentation results, (f) CS2-Net network segmentation results, (g) FR-UNet network segmentation results, (h) segmentation results of this invention). The first row shows the segmentation results of this invention and the comparison methods for small vessels in low-contrast retinal images; the second row shows the segmentation results of this invention and the comparison methods for small vessels in lesion areas; the third and fourth rows show the segmentation results of this invention and the comparison methods for small vessels under low contrast and lesion interference. Comparing the labeled image (Ground Truth), it can be seen from the positions indicated by the yellow arrows in the image that the segmentation results of the comparison methods clearly show missing, broken, or false detections, while the present invention segments the small vessels more accurately.

[0128] Experimental results show that the multi-scale dilated convolutional residual network model for retinal vessel segmentation proposed in this invention can accurately segment blood vessels in retinal images. Compared with other advanced segmentation methods, this invention can more accurately segment small blood vessels in retinal images under complex conditions and has better robustness.

[0129] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in this application, based on the technical solution and inventive concept of this application, should be included within the scope of protection of this application.

Claims

1. A retinal vessel image segmentation method based on a multi-scale dilated convolutional residual network, characterized in that, The method includes the following steps: Step 1: Preprocess the retinal image; Step 2: The preprocessed retinal images are augmented using random horizontal flipping, vertical flipping, random rotation with an angle range of [0, 360], and random cropping to increase the number of training images and obtain the model training set; Step 3: Improve the existing U-Net network by incorporating the characteristics of retinal images in three aspects: encoder, decoder, skip connections, and loss function. Add a multi-scale residual input (MRI) module, a multi-scale residual output (MRO) module, and a multi-scale mean pooling module. In each layer of the encoder, the MRI module processes the input image at different scales. The output of each layer's MRI module is passed to the combined dilated convolution (DC) module, which is composed of dilated convolution with different dilation rates, DropBlock, batch normalization (BN), and ReLU activation function, to obtain the output features of each layer of the encoder. Each layer of the encoder is connected to the DC module of the corresponding decoder layer through the skip connection (DRes Path) module. Except for the first layer of the decoder, the multi-scale residual output (MRO) module of each layer of the decoder receives the features output by the corresponding decoder layer's DC module and generates output feature maps at different scales. The output feature maps at different scales are weighted and fused to obtain the retinal segmentation result image. Step 3 replaces the skip connections in U-Net with the DRes Path module, which performs feature transformation in the skip connections, thereby better fusing features and preserving important details. The DRes Path module first fuses the output features of the encoder layer with the result of dilated convolution of these features, and then feeds the fused features into the corresponding layer of the decoder. Residual connections are introduced in the DRes Path module. Step 4: Use the binary cross-entropy loss function and the Dice loss function to form a hybrid loss function. Use the training dataset obtained in Step 2 to train the network model designed in Step 3 until the hybrid loss function converges to the optimal value.

2. The retinal vessel image segmentation method based on a multi-scale dilated convolutional residual network as described in claim 1, characterized in that, The preprocessing operations in step 1 are as follows: After grayscale processing, an adaptive histogram equalization method with limited contrast and a gamma correction method are used to enhance image contrast and highlight retinal vascular structures.

3. The retinal vessel image segmentation method based on a multi-scale dilated convolutional residual network as described in claim 1, characterized in that, Step 3 introduces multi-scale information into the encoder section, and passes the input images downsampled at different scales to the multi-scale residual input MRI module to extract features at different scales. The multi-scale residual input MRI module consists of two convolutional operations. After each convolutional operation, DropBlock, batch normalization and linear rectified activation function are applied. The first layer uses dilated convolution to increase the number of image channels in order to obtain higher-level representation features. The second 1×1 convolutional operation reduces the number of channels and compresses information to reduce network complexity and improve computational efficiency, avoiding the depth explosion problem. The sequence of these two consecutive convolutional layers is performed three times on the input image, each time using a different dilation rate. Then, these three output feature maps are jointly stitched together, and the jointly stitched feature map is added pixel by pixel to the original input image through residual connections to make full use of multi-scale features and enhance feature transfer capability, resulting in the MRI module output feature map. To ensure that the output features of the MRI module match the output features of the stitched dilated convolution module, the number of channels in the MRI module increases with the network depth.

4. The retinal vessel image segmentation method based on a multi-scale dilated convolutional residual network as described in claim 1, characterized in that, Step 3: In addition to the first layer of the decoder, add a multi-scale residual output (MRO) module after the combined dilated convolution module of each layer of the decoder to extract features at different scales. The multi-scale residual output MRO module adds an upsampling method to the MRI module to ensure that the size of each MRO output feature map is consistent with the size of the 3DC block output feature map of the decoder. The outputs of the three MRO modules and the 3DC module in the decoder are all 1-channel feature maps, which are weighted and concatenated to form a 4-channel feature map, where the weighting coefficients w1~w4 take values ​​from 0 to 1. Finally, this feature map is processed by applying 1×1 convolution and the Sigmoid activation function to produce a probability value image with values ​​ranging from 0 to 1.

5. The retinal vessel image segmentation method based on a multi-scale dilated convolutional residual network as described in claim 1, characterized in that, Step 3 introduces a combined dilated convolution DC module consisting of dilated convolution with different dilation rates, DropBlock, Batch Normalization (BN), and ReLU activation function. This combination structure with dilation rates of 1, 2, and 3 is named 1DC module, 2DC module, and 3DC module. These DC modules can accelerate network training convergence and effectively alleviate the overfitting problem of convolutional networks. In step 3, the improved U-Net model gradually increases the expansion rate of the DC module in the left encoder as the depth increases, while gradually decreasing the expansion rate of the DC module in the right decoder as the depth decreases.

6. The retinal vessel image segmentation method based on a multi-scale dilated convolutional residual network as described in claim 1, characterized in that, Step 3 adds a multi-scale mean pooling (MAP) module between the convolutional layers of each encoder and decoder in U-Net. The MAP module addresses the issue of varying object sizes during segmentation. The MAP module outputs... ,in c The number of image channels. h Image height, w Image width, Z The calculation method is as follows: ； ； ； in, This indicates the characteristics of the input MAP module. U Indicates an upsampling operation. For mean pooling operation, where This indicates the size of the pooling kernel, and , and [ ] represents the convolution kernel, and [ ] represents the connection operation. Represents standard convolution; The pyramid-structured MAP module encodes the input feature map S to capture global contextual information. This pyramid structure uses four receptive fields of different sizes for pooling operations. To balance the increase in parameters, after each pyramid pooling layer, the feature map undergoes dimensionality reduction via a 1×1 convolution, reducing the number of channels to 1 / 3 of the original. N , N The pyramid layer number is used, and then the 1×1 convolution result is upsampled using bilinear interpolation. Finally, the input features are concatenated with all upsampled feature maps to obtain the final result. ;However There are semantic incompatibility issues between different feature maps, and direct fusion may lead to inaccurate segmentation. To effectively fuse features, the MAP module introduces another 1×1 convolutional layer. This convolutional layer reduces feature dimensionality while achieving effective feature fusion.

7. The retinal vessel image segmentation method based on a multi-scale dilated convolutional residual network as described in claim 1, characterized in that, In step 3, the U-Net network uses the binary cross-entropy loss function L. bce and Dice loss function L dice Composed of a hybrid loss function, the binary cross-entropy loss function L bce and Dice loss function L dice The calculations are shown in the following formulas: ； ； in These are model predictions. Its value reflects the pixel Probability of being predicted as a blood vessel pixel: the larger the value, the greater the probability; It is a label, and its value is 0 or 1; n Let be the number of pixels in the image; the total loss function is expressed as: 。 8. The retinal vessel image segmentation method based on a multi-scale dilated convolutional residual network as described in claim 1, characterized in that, Step 3: During model training, the Adam optimizer is used with a learning rate of 0.000001 to 0.001, 150 to 250 training iterations, a batch size of 4 to 8, a DropBlock size of 4 to 7 for each dataset, and the output probability of each neuron is kept at 0.1 to 0.9.

Images