Retinal fundus blood vessel segmentation method based on LBP operator and double U-shaped network

Abstract

The application provides a retinal fundus blood vessel segmentation method based on an LBP operator and a double-U network, and comprises the following steps: acquiring a public expert-annotated color fundus blood vessel segmentation dataset, which is divided into a training set, a verification set and a test set for network training and testing; performing data preprocessing on an original color fundus image, calculating an LBP code image on a grayscale image by using an LBP operator, and performing data augmentation by means of random slicing; constructing a double-U fundus blood vessel segmentation network with a double-branch residual decoding block, including a high-level semantic network taking the preprocessed fundus image as input, a shallow texture network taking the LBP code image as input, and a feature fusion module fusing high-level semantic features extracted by the high-level semantic network and fine-grained texture features extracted by the shallow texture network; obtaining an optimal segmentation model through multiple rounds of iteration; and verifying the segmentation effect and generalization performance of the model. The application improves the prediction ability of the semantic segmentation model and improves the generalization performance of the model.

Description

Technical Field

[0001] This invention belongs to the field of medical image analysis and processing technology, and in particular relates to a method for retinal fundus vessel segmentation based on LBP operator and double U-shaped network. Background Technology

[0002] In clinical medical diagnosis, changes in the distribution and morphology of retinal vessels are closely related to many eye diseases and complex conditions, such as diabetic retinopathy, age-related macular degeneration, glaucoma, hypertension, and coronary heart disease. Retinal vessel segmentation is highly beneficial for the prevention, early diagnosis, and development of treatment plans for these diseases, and has significant clinical implications. With the rapid development of deep learning, fundus vessel segmentation methods based on convolutional neural networks have achieved excellent performance. However, due to the typically small size of pixel-level labeled datasets used for fundus vessel segmentation, the highly complex morphological structure of fundus vessels, the extreme imbalance in pixel counts between thick and thin vessels, and the very low contrast between thin vessels, especially capillaries, and the tendency of deep networks to smooth and blur local details, problems such as blurred vessel boundaries and loss of small vessels are easily encountered. Furthermore, the performance of existing methods often degrades significantly across datasets, failing to generalize well to fundus vessel images with different appearances and making them difficult to apply to real-world environments. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a retinal fundus vessel segmentation method based on the LBP operator and a dual U-shaped network. This method utilizes the LBP code image obtained by LBP transformation of the fundus grayscale image to capture fine-grained features, restore boundaries and details, and improve the model's generalization ability. At the same time, it uses a dual-branch decoding block with residuals to reduce the false detection rate and further improve the model's performance.

[0004] Local Binary Pattern (LBP) is a classic, traditional handcrafted feature, proposed by Ojala et al. in 1994. The LBP operator is defined within a 3×3 neighborhood. Using the center pixel as a threshold, it compares the grayscale values ​​of its eight neighboring pixels. If the values ​​of surrounding pixels are greater than the center pixel's value, the pixel is marked as 1; otherwise, it is marked as 0. This comparison of the eight points in the neighborhood produces an 8-bit binary number (usually converted to decimal, i.e., the LBP code, of which there are 256 possible values), which is the LBP value of the center pixel in the neighborhood. This value is used to reflect the texture information of the region. The LBP operator is simple and efficient to compute, aggregates the relationships between pixels in the neighborhood, and is robust to changes in illumination. It also describes fine-grained texture information well, so it was widely used in early applications such as face recognition and object detection.

[0005] This invention provides a retinal fundus vessel segmentation method based on the LBP operator and a dual U-shaped network. It introduces the traditional LBP operator to compensate for the deficiencies of pixel-level features extracted by deep networks, improve the predictive ability of semantic segmentation models, and enhance the generalization performance of the models.

[0006] The technical solution adopted in this invention provides a retinal vessel segmentation method based on the LBP operator and a dual U-shaped network, comprising the following steps:

[0007] Step 1: Obtain a publicly available color fundus vessel segmentation dataset annotated by experts, and divide it into training set, validation set, and test set for network training and testing;

[0008] Step 2: Perform data preprocessing on the original color fundus image and use the LBP operator to calculate the LBP code image on the grayscale image;

[0009] Step 3: Data augmentation is performed on the preprocessed fundus images and corresponding vascular annotations and LBP code images by random slicing;

[0010] Step 4: Construct a dual U-shaped network, which includes two sub-U-shaped networks and a feature fusion module. The sub-networks are a high-level semantic network and a shallow texture network. The preprocessed fundus image and LBP code image are input into the high-level semantic network and the shallow texture network, respectively. The pixel-level high-level semantic features extracted by the high-level semantic network and the fine-grained texture features extracted by the shallow texture network are input into the feature fusion module to generate a pixel-level prediction probability map and calculate the segmentation loss function.

[0011] Step 5: Based on Step 4, complete multiple rounds of network training iterations to obtain the optimal segmentation model;

[0012] Step 6: Verify the model's segmentation performance on the test set;

[0013] Step 7: Validate the model’s generalization performance on several other publicly available fundus datasets.

[0014] Furthermore, in step 2, the LBP transform function is as follows:

[0015]

[0016] Among them, (x c ,y c () represents the center element in a 3×3 neighborhood, with a pixel value of i. c i p represents the value of other pixels in the neighborhood, p is the number of pixels in the neighborhood, and s is the sign function.

[0017]

[0018] Furthermore, in step 4, the feature fusion module corresponds to a weighted addition operation, which is here extended to a learnable and more flexible approach, namely,

[0019] F = Conv([F S ,F L ])

[0020] Where F S F represents pixel-level high-level semantic features extracted by a high-level semantic network. L Fine-grained texture features are extracted from a shallow texture network. [·,·] denotes channel-level concatenation, and Conv is a 1×1 convolution. First, high-level semantic features are concatenated with fine-grained texture features. Then, a simple and effective 1×1 convolution is used to achieve channel-level feature fusion, resulting in the detail-enhanced feature F. Note that F... s With F L They should have the same shape. Otherwise, a bilinear interpolation upsampling operation needs to be performed on the smaller one to align the shapes.

[0021] Furthermore, in step 4, the high-level semantic network is the original U-Net with DropBlock; the shallow texture network is a small U-Net with DropBlock.

[0022] The original U-Net is a five-layer encoder-decoder architecture with skip connections. The encoder consists of five convolutional blocks, each containing two 3×3 convolutional layers, a BN layer, and a ReLU layer. Downsampling layers are between the convolutional blocks. The decoder consists of four convolutional blocks with the same structure as the encoder. Upsampling layers are between the convolutional blocks. The features of the convolutional blocks in the encoder are upsampled to the same resolution by the corresponding convolutional blocks in the decoder through skip connections, concatenated, and input into the next convolutional block of the decoder.

[0023] The original U-Net with DropBlock adds DropBlock after each 3×3 convolutional layer to further prevent overfitting. The smaller U-Net with DropBlock reduces the number of channels per layer of the original U-Net with DropBlock by 8 times.

[0024] Furthermore, the U-Net decoder with DropBlock replaces the four convolutional blocks with four dual-branch decoder blocks with residuals. Specifically, each dual-branch decoder block with residuals contains one residual and two branches. The residual branch contains a 1×1 convolutional layer and a batch normalization (BN) layer. The first branch contains a 3×3 convolutional layer and a BN layer, and the second branch contains two 3×3 convolutional layers and a BN layer. Between the first BN layer and the second 3×3 convolutional layer is a LeaklyReLU layer. The outputs of the first and second branches are concatenated and added to the output of the residual branch. Finally, a LeaklyReLU layer is passed through to obtain the final output.

[0025]

[0026] in f(·), g(·), σ(·), and [·,·] represent the residual branch, the first branch, the second branch, LeaklyReLU, and splicing, respectively.

[0027] Furthermore, in step 4, the segmentation loss function is the BCE (Binary Cross Entropy) loss function, that is,

[0028]

[0029] Where y represents the real label. For predicted labels.

[0030] Furthermore, in step 5, the optimal model is obtained by determining whether the current model is the current optimal model based on the validation set effect during the training process. If it is, it will be overwritten and saved until the specified number of iterations is completed.

[0031] The beneficial effects of this invention are as follows:

[0032] (1) To address the issues of blurred boundaries and degraded details, the traditional LBP operator is introduced into the deep learning network, which can capture local texture information simply and efficiently. Considering the limited information content of LBP code images and the potential for overfitting in deep models, a shallow texture network is used to extract fine-grained texture features;

[0033] (2) A dual-U-shaped network structure was designed, capable of simultaneously extracting high-level semantic features and fine-grained texture features. Considering that the fine-grained texture features extracted by shallow texture networks may contain unwanted textures and noise, a feature fusion module was designed to restore details while eliminating adverse effects on the segmentation results. A dual-branch decoding block with residuals was designed to further reduce false positives and prevent false detections.

[0034] (3) This invention can be trained in an end-to-end manner, requiring only the LBP code image to be calculated in advance, thus improving the segmentation effect of the model in a simple and effective way.

[0035] (4) Since the LBP operator aggregates the intensity relationship between the center point and the surrounding points and has grayscale invariance, the model has better segmentation stability for image data of different fundus image datasets, which improves the generalization of the segmentation model. Attached Figure Description

[0036] Figure 1 This is a flowchart of an embodiment of the present invention.

[0037] Figure 2 From top to bottom, the images are the original image, the preprocessed image, and an example of an LBP code image according to an embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram of the network structure of the present invention. Detailed Implementation

[0039] To facilitate understanding and implementation of the present invention by those skilled in the art, a further detailed description is provided below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0040] See Figure 1 The present invention provides a method for retinal fundus vessel segmentation based on LBP operator and double U-shaped network, comprising the following steps:

[0041] Step 1: Obtain a publicly available, expert-annotated color fundus vessel segmentation dataset, and select a certain proportion of image samples {1,…,} from the fundus image dataset. n}, where n is the number of image samples and the label y = [y1, ..., ... n One sample is used as the training sample, and the rest are used as the test sample.

[0042] In this embodiment, the DRIVE dataset is used, which contains 40 color retinal fundus images and corresponding fundus vessel segmentation labels, of which 20 images are the training set and 20 images are the test set. Specifically, each image in the test set is labeled by two doctors, and in this embodiment, the label of the first doctor is used as the ground truth label.

[0043] Step 2: Preprocess the training samples. Optionally, in this embodiment, the original color fundus image is subjected to grayscale conversion, standardization, contrast-limited adaptive histogram equalization (CLAHE), and GAMMA correction to enhance contrast, resulting in... And calculate LBP code image Where ψ(·) represents the LBP transform function. Specifically, within a 3×3 neighborhood, using the center pixel as a threshold, the gray values ​​of the eight adjacent pixels are compared with it. If the value of the surrounding pixels is greater than the value of the center pixel, the position of that pixel is marked as 1; otherwise, it is marked as 0. After comparing the eight points in the neighborhood, starting from the top-left pixel and proceeding clockwise, an 8-bit binary number is obtained. Converting this to decimal gives the LBP code of the center pixel in that neighborhood.

[0044] Because it aggregates the intensity relationship between the center pixel and the surrounding pixels, LBP code images can characterize boundary context changes and detailed features, and have grayscale invariance. Figure 2 The example image is obtained by preprocessing and LBP transformation of fundus images from the DRIVE dataset.

[0045] Step 3: Since fundus vessel datasets are typically small, data augmentation is performed using random slicing. In this example, the DRIVE training set contains only 20 fundus images. The training samples are randomly cropped into 64×64 patches (image blocks), with each patch obtained by randomly selecting the center of the patch within the entire image. A total of 150,000 images are used, with 90% used for training and 10% for validation. The batch size is set to 64, representing the number of training samples input into the network per cycle. Furthermore, data augmentation such as random cropping (48×48 size), random flipping, and random rotation are applied to the training patches. During the testing phase, an overlapping tiling strategy is used to generate the final segmentation results, with the test set patch size being 96×96.

[0046] Step 4: Construct a dual-U-shaped fundus vessel segmentation network, comprising a high-level semantic network that takes preprocessed fundus images as input, a shallow texture network that takes LBP code images as input, and a feature fusion module that integrates high-level semantic features extracted by the high-level semantic network and fine-grained texture features extracted by the shallow texture network. The U-shaped network is a classic encoder-decoder structure with skip connections, widely used in fundus vessel segmentation. See also... Figure 3This embodiment uses the original U-Net with DropBlock as the high-level semantic network. Since LBP code images contain local texture information, especially the boundaries of some blood vessels and small blood vessels, but due to limited information, to prevent overfitting of the deep model, this embodiment uses a small U-Net with DropBlock as a shallow texture network. Specifically, the original U-Net is a five-layer encoder-decoder structure with skip connections. The encoder includes five convolutional blocks, each containing two 3×3 convolutional layers, a BN layer, and a ReLU layer. Downsampling layers connect the convolutional blocks. The decoder includes four convolutional blocks with the same structure as the encoder, connected by upsampling layers. The features of the convolutional blocks in the encoder are upsampled to the same resolution by skip connections and then concatenated with the corresponding convolutional blocks in the decoder before being input into the next layer of the decoder. In this embodiment, the high-level semantic network adds DropBlock after each 3×3 convolutional layer to further prevent overfitting. The block size is set to 3, and the first layer has 64 output channels. The small U-Net reduces the number of channels per layer by a factor of 8, resulting in 8 channels in the first layer. While LBP significantly improves the detection rate of small blood vessels, it also introduces false positives. Therefore, to reduce the false positive rate, this embodiment constructs each layer of the U-Net decoder's convolutional block as a dual-branch decoding block with residuals. Specifically, this decoding block contains one residual and two branches. The residual branch contains a 1×1 convolutional layer and a batch normalization (BN) layer. The first branch contains a 3×3 convolutional layer and a BN layer, and the second branch contains two 3×3 convolutional layers and a BN layer. A LeaklyReLU layer separates the first BN layer from the second 3×3 convolutional layer. The outputs of the first and second branches are concatenated and added to the output of the residual branch. Finally, a LeaklyReLU layer is applied to obtain the final output of the residual convolutional block. That is,

[0047]

[0048] in f(·), g(·), σ(·), and [·,·] represent the residual, the first branch, the second branch, LeaklyReLU, and splicing, respectively.

[0049] Two U-shaped fundus vessel segmentation networks with dual-branch residual decoding blocks can simultaneously extract high-level semantic features and fine-grained texture features. The two sets of features are then input into a feature fusion module. Specifically, the two sets of features are first concatenated, and then a 1×1 convolution is applied to obtain details-enhanced features. The feature fusion module can compensate for the shortcomings of blurred boundaries and loss of small blood vessels in high-level semantic features, while reducing the impact of texture and noise in fine-grained texture features on the segmentation results. The feature fusion module corresponds to a weighted addition operation; here, it is extended to a learnable and more flexible approach.

[0050] F = Conv([F S ,F L ])

[0051] Where F S F represents pixel-level high-level semantic features extracted by a high-level semantic network. L Fine-grained texture features are extracted from a shallow texture network. [·,·] denotes channel-level concatenation, and Conv is a 1×1 convolution. First, high-level semantic features are concatenated with fine-grained texture features. Then, a simple and effective 1×1 convolution is used to achieve channel-level feature fusion, resulting in the detail-enhanced feature F. Note that F... S With F L They should have the same shape. Otherwise, a bilinear interpolation upsampling operation needs to be performed on the smaller one to align the shapes.

[0052] Next, the BCE loss is calculated, that is,

[0053]

[0054] Where y represents the real label. For predicted labels.

[0055] Step 5: During training, determine whether the current model is the optimal model based on the validation set results. If so, it will be overwritten and saved until the specified number of iterations is completed. Optionally, in this embodiment, in each round, compare whether the AUC metric of the current validation set is the highest. If so, the current model is the optimal model, and it is overwritten and saved. After 50 rounds of training iterations, the optimal segmentation model is obtained.

[0056] Step 6: Verify the model segmentation effect. Calculate commonly used fundus vessel segmentation indices such as AUC and F1 on the test set. Of particular note is the SE (Sensitivity) index, which represents the segmentation effect of fine vessels.

[0057] The implementation example, by applying the model to the DRIVE test set, shows improvements in AUC, F1, ACC (Accuracy), SE (Sensitivity), and SP (Specificity) compared to U-Net, by 1.2%, 1.42%, 1.25%, 4.84%, and 0.2%, respectively. Particularly for SE (Sensitivity), which reflects the effectiveness of fine vessel segmentation, and SP (Specificity), which reflects false positives, using only the original U-Net convolutional blocks and LBP feature fusion resulted in a 5.9% improvement in SE, but a 1.77% decrease in SP. Adding the constructed two-branch residual decoding block slightly reduced the SE improvement to 4.84%, but SP increased by 0.2%, and all other segmentation metrics also improved.

[0058] Step 7: Validate the model’s generalization performance by testing it on other publicly available color fundus image blood vessel segmentation datasets that were not seen during training.

[0059] The example demonstrates the cross-dataset segmentation performance of the model trained with DRIVE on the STARE and CHASE_DB1 datasets. On the STARE dataset, AUC and F1 scores improved by 1.59% and 3.04%, respectively, and SE improved by 11.16%. On the CHASE_DB1 dataset, AUC and F1 scores improved by 2.69% and 10.57%, respectively, and SE improved by 19.95%.

[0060] It should be understood that all parts not described in detail in this specification belong to the prior art. The above description of the preferred embodiments is relatively detailed, but it should not be regarded as a limitation on the scope of protection of this invention. Those skilled in the art can make substitutions or modifications under the guidance of this invention without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for retinal and fundus vessel segmentation based on the LBP operator and a double U-shaped network, characterized in that, Includes the following steps: Step 1: Obtain a publicly available color fundus vessel segmentation dataset annotated by experts, and divide it into training set, validation set, and test set for network training and testing; Step 2: Perform data preprocessing on the original color fundus image and use the LBP operator to calculate the LBP code image on the grayscale image; Step 3: Data augmentation is performed on the preprocessed fundus images and corresponding vascular annotations and LBP code images by random slicing; Step 4: Construct a dual U-shaped network. The dual U-shaped network includes two sub-U-shaped networks and a feature fusion module. The sub-networks are a high-level semantic network and a shallow texture network. The preprocessed fundus image and LBP code image are input into the high-level semantic network and the shallow texture network, respectively. The pixel-level high-level semantic features extracted by the high-level semantic network and the fine-grained texture features extracted by the shallow texture network are input into the feature fusion module to generate a pixel-level prediction probability map. The segmentation loss function is then calculated with the real label. The high-level semantic network is the original U-Net with DropBlock; the shallow texture network is a small U-Net with DropBlock. The original U-Net is a five-layer encoder-decoder architecture with skip connections. The encoder consists of five convolutional blocks, each containing two 3×3 convolutional layers, a BN layer, and a ReLU layer. Downsampling layers are between the convolutional blocks. The decoder consists of four convolutional blocks with the same structure as the encoder. Upsampling layers are between the convolutional blocks. The features of the convolutional blocks in the encoder are upsampled to the same resolution by the corresponding convolutional blocks in the decoder through skip connections, concatenated, and input into the next convolutional block of the decoder. The original U-Net with DropBlock adds DropBlock after each 3×3 convolutional layer to further prevent overfitting. The smaller U-Net with DropBlock reduces the number of channels per layer of the original U-Net with DropBlock by 8 times. The U-Net decoder with DropBlock replaces the four convolutional blocks with four dual-branch decoder blocks with residuals. Specifically, each dual-branch decoder block with residuals contains one residual and two branches. The residual branch contains a 1×1 convolutional layer and a batch normalization (BN) layer. The first branch contains a 3×3 convolutional layer and a BN layer, and the second branch contains two 3×3 convolutional layers and a BN layer. A LeaklyReLU layer separates the first BN layer from the second 3×3 convolutional layer. The outputs of the first and second branches are concatenated and added to the output of the residual branch. Finally, a LeaklyReLU layer is passed through to obtain the final output. in , , , , These are the residual branch, the first branch, the second branch, LeaklyReLU, and the concatenation; Step 5: Based on Step 4, complete multiple rounds of network training iterations to obtain the optimal segmentation model; Step 6: Verify the model's segmentation performance on the test set; Step 7: Validate the model’s generalization performance on several other publicly available fundus datasets.

2. The retinal fundus vessel segmentation method based on LBP operator and double U-shaped network according to claim 1, characterized in that: In step 2, the LBP transform function is as follows: in, express The center element in the neighborhood has a pixel value of , Represents the value of other pixels in the neighborhood. The number of pixels in the neighborhood. For symbolic functions, 。 3. The retinal fundus vessel segmentation method based on LBP operator and double U-shaped network according to claim 1, characterized in that: In step 4, the processing procedure of the feature fusion module is as follows: in Pixel-level high-level semantic features extracted from high-level semantic networks Fine-grained texture features extracted from shallow texture networks. This indicates channel-level splicing. The process involves a 1×1 convolution; first, high-level semantic features are concatenated with fine-grained texture features, then a simple and effective 1×1 convolution is used to achieve channel-level feature fusion, resulting in the detail-enhanced feature F; where... and If they have the same shape, otherwise, perform a bilinear interpolation upsampling operation on the smaller one to align the shapes.

4. The retinal fundus vessel segmentation method based on LBP operator and double U-shaped network according to claim 1, characterized in that: The segmentation loss function is the BCE loss function, that is, Where y represents the real label. For predicted labels.

5. The retinal fundus vessel segmentation method based on LBP operator and double U-shaped network according to claim 1, characterized in that: In step 5, the optimal segmentation model is obtained by determining whether the current model is the current optimal model based on the validation set performance during the training process. That is, whether the AUC index of the current validation set is the highest. If so, it will be overwritten and saved until the specified number of iterations is completed.

6. The retinal fundus vessel segmentation method based on LBP operator and double U-shaped network according to claim 1, characterized in that: In step 6, the model's segmentation performance is verified by calculating the metrics AUC, F1, ACC (Accuracy), SE (Sensitivity), and SP (Specificity).

Images