Diabetic retinopathy segmentation method based on multi-branch and visual manhattan

Abstract

The diabetes retinopathy segmentation method based on multi-branch and visual mamba relates to the field of medical image processing.The existing diabetes retinopathy segmentation method still has the problems of difficulty in accurately segmenting small target lesions,complex model and insufficient modeling ability of global context information.The diabetes retinopathy segmentation model is provided,which takes a color fundus image as input data,extracts multi-scale information through a multi-branch multi-scale attention fusion block MSAF Block,obtains effective context information by using a mamba fusion bottleneck block MSMFB of different scales,constructs a joint weighted jump module by a shared attention jump connection module to effectively fuse multi-level features,finally outputs the final segmentation result through a lightweight decoder.The model can realize efficient segmentation of four kinds of lesions.The method is also suitable for the field of diabetes retinopathy image processing.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a segmentation method for diabetic retinopathy based on multi-branch and visual Mamba. Background Technology

[0002] Diabetic retinopathy is a typical microvascular complication of diabetes and a major cause of vision impairment and blindness in adults. Currently, clinical diagnosis primarily relies on color fundus imaging. Traditional diagnostic methods suffer from high subjectivity, low efficiency, and high cost.

[0003] In recent years, deep learning technology has been widely proven to have excellent performance in various computer vision tasks. Particularly in the medical field, it can produce results with high accuracy and discriminative power. However, existing methods for diabetic retinopathy still face challenges such as difficulty in accurately segmenting small lesions, model complexity, and insufficient ability to model global contextual information. Summary of the Invention

[0004] To overcome the problems of inaccurate segmentation of small lesions, complex models, and insufficient modeling ability of global context information in existing technologies for diabetic retinopathy, this invention proposes an end-to-end lightweight and efficient deep learning model, namely the Multi-scale Attention Convolution Mamba Network. This invention achieves the above technical problems through the following technical solutions:

[0005] Option 1: This invention proposes a segmentation method for diabetic retinopathy based on multi-branch and visual Mamba, the method comprising the following steps:

[0006] Step 1: Obtain images of diabetic retinopathy and pixel-level labeled image data of microaneurysms, hard exudates, soft exudates, and hemorrhages from the Indian diabetic retinopathy image dataset, and preprocess the pixel-level labeled image data.

[0007] Step 2: Divide the diabetic retinopathy images obtained in Step 1 into a training set and a test set;

[0008] Step 3: Construct an MSAF Block model based on the diabetic retinopathy images segmented in Step 2 to extract local texture features, edge details, and small-scale spatial structures, thereby enhancing the MSAF Block model's ability to perceive fine-grained targets such as small lesions.

[0009] Step 4, based on the local texture features of diabetic retinopathy obtained in step 3, a Mamba fusion bottleneck model MSMFB is constructed to capture long-range dependencies and spatial context associations and obtain global semantic information of the lesions;

[0010] Step 5, based on the texture features of diabetic retinopathy obtained in step 3, a Joint-Weighted Skip Module is constructed to fuse lesion information between different layers;

[0011] Step 6, based on the enhanced lesion features obtained from the Mamba fusion bottleneck model MSMFB constructed in step 4 and the Joint-Weighted Skip Module constructed in step 5, an attention-guided lightweight decoder is constructed to complete the segmentation task of diabetic retinopathy;

[0012] Step 7, the Multi-scale Attention Convolution Mamba Network is trained and verified on the Indian diabetic retinopathy image dataset described in step 1 to obtain a diabetic retinopathy segmentation model, and the diabetic retinopathy segmentation based on multi-branch and visual Mamba is completed.

[0013] Further, a preferred embodiment is provided, wherein the preprocessing in step 1 includes the steps of reading the image, label grayscale, random rotation and flipping, random cropping to 256x256, converting the numpy data type, and adjusting the channel dimension.

[0014] Further, a preferred embodiment is provided, wherein the method for dividing the diabetic retinopathy images obtained in step 1 into training set and test set in step 2 is to process the Indian diabetic retinopathy image dataset in step 1 by using five-fold cross-validation, and the number of images in the training set and test set is 54 and 27 respectively.

[0015] Further, a preferred embodiment is provided, wherein the method for constructing the MSAF Block model in step 3 is:

[0016] The preprocessed diabetic retinopathy images in step 1 are randomly divided into sub-regions with a size of 256x256, and the Multi-Path Attention Block and Dual-Path Cross Attention Fusion Block are executed on the feature matrix of each sub-region.

[0017] Further, a preferred embodiment is provided, and a method for performing a Multi-Path Attention Block operation is as follows: the Multi-Path Attention Block includes parallel attention convolution modules and multi-level down-sampling modules; the attention convolution module is used to combine the coordinate attention mechanism with the residual connection; the multi-level down-sampling module is used to extract features through multi-level pooling down-sampling and a series of convolution operations, and the obtained features are up-sampled to the original spatial scale level by level.

[0018] Further, a preferred embodiment is provided, and a method for performing a Dual-Path Cross Attention Fusion Block operation on the obtained feature matrix is as follows: the obtained multi-scale features are input into a dual-branch fusion module, and the final encoding output features are generated through a multi-scale pooling and spatial attention parallel strategy.

[0019] Further, a preferred embodiment is provided, and a method for constructing the MSMFB model in step 4 is as follows:

[0020] The MSMFB model includes a multi-scale dilated convolution, an SS2D in the Vmamba framework, and a global-local feature fusion module, wherein the SS2D is used to capture the global context information across the region, obtain the overall semantic relationship through local convolution, and improve the recognition ability of large-scale lesion regions or sparse lesion regions;

[0021] The global-local feature fusion module is used to perform weighted processing on the input features based on the corresponding weight function, and perform feature combination on the weighted features to form a joint representation; and generate a global weight for adjusting the overall feature response according to the joint representation, and apply it to the initial input to obtain the final weighted feature output.

[0022] Further, a preferred embodiment is provided, and a method for constructing a Joint-Weighted Skip Module in step 5 is as follows:

[0023] The input multiple feature maps are multiplied by the weights obtained by the sigmoid function to perform weighted and fused, and a feature fusion layer is constructed to fuse multi-level feature information; the feature map output after fusing the multi-level feature information is output to complete the construction of the Joint-Weighted Skip Module.

[0024] Scheme II, a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to realize the steps of the method of scheme I.

[0025] Scheme three, a computer device, comprising a memory and a processor, the memory has stored therein a computer program, when the processor runs the computer program stored in the memory, the processor executes the method of scheme one.

[0026] The present application has the advantages of:

[0027] The diabetes retinopathy segmentation method based on multi-branch and visual mamba provided by the present application combines the Multi-Path Attention Block and the Dual-Path Cross-Attention Fusion Block, can effectively integrate local and global information, fully play the complementary effect of the two, and thus improve the expression ability and accuracy of the model.

[0028] The MSAF Block provided by the present application adopts a multi-scale feature extraction mechanism, through operations such as small sampling pooling, so that the model can capture multi-scale information in the image, thereby enhancing the expression ability of the features. Meanwhile, a two-branch shared attention mechanism is designed, under the premise of ensuring efficient calculation, the multi-scale features are combined for weighted processing, so as to realize more accurate feature fusion and information transmission.

[0029] The MSMFB provided by the present application can effectively capture macro information in the image by introducing a global feature extraction mechanism. Such global features can help the model understand the overall structure and context of the image, thereby enhancing the recognition ability of complex patterns.

[0030] The Joint-Weighted Skip Module provided by the present application combines features of different layers through a multi-level feature fusion mechanism, effectively integrates information of different scales, and thus improves the expression ability of the model on complex image content. Meanwhile, it can capture deeper spatial dependency relationships and complex cross-level feature associations.

[0031] The present application is suitable for the field of early diagnosis of diabetic retinopathy and segmentation of diabetic retinopathy. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The flowchart of the diabetes retinopathy segmentation method based on multi-branch and visual mamba according to the first embodiment.

[0033] Figure 2 The flowchart of the pre-processing according to the first embodiment. DETAILED DESCRIPTION

[0034] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application.

[0035] Embodiment one, see Figure 1 To illustrate the present embodiment, the present embodiment proposes a diabetic retinopathy segmentation method based on multi-branch and visual mamba, which specifically comprises the following steps:

[0036] Step 1, obtain diabetic retinopathy images and pixel-level annotations of four lesions including microaneurysms, hard exudates, soft exudates and hemorrhages from the Indian diabetic retinopathy image dataset, and preprocess the above data. Based on the preprocessed images and labels, a dataset for diabetic retinopathy segmentation task is constructed.

[0037] Step 1.1, download color fundus images containing lesion regions and corresponding label images from the Indian diabetic retinopathy image dataset.

[0038] Step 1.2, fundus image preprocessing includes the following steps:

[0039] Step 1.2.1, the first part of preprocessing, reading the image and converting it to a grayscale image;

[0040] Step 1.2.2, the second part of preprocessing, randomly rotating the image with an angle range of-30° to 30°;

[0041] Step 1.2.3, the third part of preprocessing, randomly flipping the image to enhance the robustness of the model;

[0042] Step 1.2.4, the fourth part of preprocessing, randomly cropping the image to 256x256 size to keep the input size consistent and increase the diversity of training data;

[0043] Step 1.2.5, the fifth part of preprocessing, converting the image to NumPy array format for subsequent processing and calculation;

[0044] Step 1.2.6, the sixth part of preprocessing, adjusting the channel dimension of the image to meet the requirements of model input, ensuring that each image has the correct number of channels.

[0045] Step 2, divide the diabetic retinopathy images obtained in step 1 into training set and test set. The number of pictures in the training set and test set is 54 and 27 respectively.

[0046] Step 3, based on the diabetic retinopathy image divided in step 2, the MSAF Block model is constructed, which is used to extract local texture features, edge details and small scale spatial structure, so as to enhance the perception ability of the model to the fine-grained target such as micro lesions. The construction steps of MSAF Block model are as follows:

[0047] Step 3.1, import subunit, used to construct the basic library required by MSAF Block model: including torch, torch.nn and torch.nn.functional, and introduce custom ConvAtt_Block, ReduceConv, ReduceConvUp, DualBranchModule and other submodules.

[0048] Step 3.2, define ConvAtt_Block submodule, used to construct convolution attention module: the module contains depth separable convolution, channel attention mechanism, batch normalization, activation function, feature expansion and compression, and residual connection and other technologies. Through these operations, the module not only can efficiently extract image features and improve computational efficiency, but also can enhance the expression ability of the model, especially in complex tasks, which can better handle the nonlinear relationship and spatial information between features.

[0049] Step 3.3, define down-sampling convolution operation, used for multi-scale feature extraction: MaxPool2d is used for pooling operation to reduce the spatial size of feature map, and ReduceConv and ReduceConvUp are used for channel number adjustment and up-sampling operation to ensure that the feature map can effectively transmit and fuse multi-scale information at each stage.

[0050] Step 3.4, define DualBranchModule feature submodule based on step 3.2 and step 3.3, used to construct feature fusion module in encoder: the module further refines the expression of image features through the interaction of features in two different branches. Each branch first goes through its own multi-scale pooling and channel attention mechanism to get enhanced features, and then goes through the multi-scale pooling and channel attention mechanism of the other branch to perform feature weighting and enhancement.

[0051] Step 3.5, define forward propagation process: in the forward propagation process, first pass through the initial convolution conv0 to extract features, then gradually extract multi-scale features through multiple pooling, convolution and up-sampling operations. The features at each stage are fused with the features at the previous stage to ensure that the information is effectively transmitted at multiple levels. Finally, the fused features are output through DualBranchModule.

[0052] Step 4, Based on the texture features of diabetic retinopathy obtained in step 3, MSMFB is constructed to capture long-range dependencies and spatial context associations, enabling the model to obtain global semantic information of the lesions.

[0053] Step 4.1, Import subunit, used as the base library for MSMFB module, including torch, torch.nn, and custom Multi-Path Attention Block and Dual-Path Cross Attention Fusion Block.

[0054] Step 4.2, A MambaModule class is defined to process input images and extract multi-scale features. The module contains three MambaBlock, each using different convolution kernels to process input images to capture information at different scales, and further fuse features for each scale.

[0055] Step 4.3, Based on step 4.2, a FeatureFusionModule is added to fuse features at each scale. This module further enhances feature representation capabilities by weighting and combining features at different scales, enabling the model to focus on both details and global information.

[0056] Step 4.4, In forward propagation, first extract multi-scale features through MambaBlock, then fuse each scale feature with the original input and add residual connection. The fused features are further processed through FeatureFusionModule, and finally compressed using DoubleConv to obtain the final fused feature output. These features will be used for subsequent image segmentation tasks.

[0057] Step 5, Based on the texture features of diabetic retinopathy obtained in step 3, Joint-WeightedSkip Module is constructed to fuse lesion information between different layers.

[0058] Step 5.1, Import subunit, used as the base library for constructing Joint-Weighted Skip Module, including torch, torch.nn.

[0059] Step 5.2, a SigmoidEnhance3 class is defined to weight and fuse multiple feature maps input. In the initialization, three convolutional layers are set up, each using a 3x3 convolutional kernel and the same number of channels to extract the weight of the input feature map. Through the sigmoid activation function, the feature map of the convolution output is mapped to the range of [0, 1] as the weighting coefficient.

[0060] Step 5.3, based on step 5.2, a feature fusion layer is added to fuse multi-level feature information. This layer uses a 1x1 convolutional kernel to fuse the three weighted feature maps. This step helps to integrate different input features and improve the model's ability to express different scale features.

[0061] Step 5.4, in the forward propagation, first calculate the weighting coefficients w1, w2 and w3 for the input feature maps x1, x2 and x3 through three convolutional layers, and weight the corresponding input feature maps to get out1, out2 and out3. Then, the three weighted feature maps are spliced and sent to the fuse layer for further fusion to get the fused feature F. Finally, use w4 (the weight calculated from F) to weight each feature map, and output the weighted out1, out2 and out3 as the final result.

[0062] Step 6, based on the MSMFB model of step 4 and the Joint-Weighted Skip Module of step 5, the enhanced lesion features are obtained, and the attention-guided lightweight decoder is constructed to complete the diabetic retinopathy segmentation task.

[0063] Step 6.1, import subunits for training and verification base library: including torch, torch.nn, torch.utils.data, numpy, pandas, nibabel, etc., and introduce the SEBNetmixcamamba class as the core model for training.

[0064] Step 6.2, define the FundusSeg_Loader dataset class to load fundus image and its corresponding label data: the path of the dataset is dynamically set according to the dataset_name parameter, and the image data is loaded using PyTorch's DataLoader. By reading the image file and preprocessing, the label data is converted to the required format.

[0065] Step 6.3. Define the train_net function for training the model and performing the validation process: During training, images and labels are loaded through train_loader, and during the training loop, input data is forward propagated, cross-entropy loss function is calculated, and model parameters are updated through backpropagation. After each epoch, validation is performed, and the accuracy and loss of the model are calculated during the validation process.

[0066] Step 6.4. Data loading and optimizer setup: In train_net, first create training and validation datasets using FundusSeg_Loader and batch load them using torch.utils.data.DataLoader. Then initialize the optimizer Adam and define the learning rate scheduler MultiStepLR to achieve learning rate decay.

[0067] Step 6.5. Define the model training and validation process: In each training epoch, the model is forward propagated, loss is calculated and backpropagation is performed to optimize the model. Every specified number of epochs, validation is performed, loss on the validation set is calculated and printed. If the loss on the validation set is less than the current best loss, the model weights are saved.

[0068] Step 6.6. Define the loss function and optimizer: During model training, the cross-entropy loss function is used for optimization. The optimizer is chosen as Adam with an initial learning rate of 0.001, and MultiStepLR is used for learning rate decay, gradually reducing the learning rate after a specified number of epochs.

[0069] Step 6.7. Loss calculation and optimization during training: Each batch of data is forward propagated through the network, the current loss is calculated and backpropagation is performed. All training data is iterated multiple times to optimize network parameters, while recording the changes in loss during training.

[0070] Step 6.8. Loss and accuracy evaluation during validation: Every 10 epochs, the model is evaluated on the validation set. The loss on the validation set and other evaluation metrics such as accuracy, recall, etc. are calculated to evaluate the performance of the model.

[0071] Step 6.9. Model weight saving and updating: If better loss or accuracy is achieved on the validation set, the model weights are saved and used in subsequent experiments.

[0072] The application takes the IDRID data set as input data, obtains local texture features and edge details through the MSAF Block, captures long-range dependence and spatial context association using the MSMFB, and obtains multi-level fusion features using the Joint-Weighted Skip Module, so as to improve the accuracy of lesion segmentation of the model.

[0073] The technical solutions of the application and the technical effects thereof will be further described below in combination with specific experimental data.

[0074] In order to verify the effectiveness of each component in the proposed model Multi-scale Attention Convolution Mamba Network, we perform an ablation experiment on the IDRID data set. As shown in Table 1, the accuracy of the final model reaches 70.36%.

[0075] Table 1 Ablation experiment

[0076]

[0077] Those skilled in the art can understand that the above description is only preferred embodiments of the application, and the features described in each embodiment and / or technical solution of the disclosure can be combined or combined, even if such combination or combination is not explicitly described in the disclosure. It is not intended to limit the application, although the application has been described in detail with reference to the foregoing embodiments, and those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacement for part of the technical features, and any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application shall be included in the protection scope of the application.

[0078] Although the preferred embodiments of the application have been described, those skilled in the art can make further changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended technical solutions are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the application. Obviously, those skilled in the art can make various modifications and changes to the application without departing from the spirit and scope of the application. Thus, if these modifications and changes of the application fall within the scope of the technical solutions and equivalents thereof, the application also intends to include these modifications and changes.

Claims

1. A method for diabetic retinopathy segmentation based on multi-branch and visual manban, characterized in that, The method comprises the following steps: Step 1, obtaining diabetic retinopathy images and pixel-level labeled image data of microaneurysms, hard exudates, soft exudates and hemorrhages from an Indian diabetic retinopathy image dataset, and preprocessing the pixel-level labeled image data; Step 2, dividing the diabetic retinopathy images obtained in step 1 into a training set and a test set; Step 3, constructing an MSAF Block model based on the diabetic retinopathy images divided in step 2, for extracting local texture features, edge details and small-scale spatial structures, i.e. enhancing the perception ability of the MSAF Block model to small lesion fine-grained targets; Step 4, constructing a MSMFB based on the local texture features of the diabetic retinopathy obtained in step 3, for capturing long-range dependencies and spatial context associations, and obtaining global semantic information of the lesions; Step 5, constructing a Joint-Weighted Skip Module based on the texture features of the diabetic retinopathy obtained in step 3, for fusing lesion information between different layers; Step 6, constructing an attention-guided lightweight decoder based on the enhanced lesion features obtained from the MSMFB constructed in step 4 and the Joint-Weighted Skip Module constructed in step 5, to complete the segmentation task of diabetic retinopathy; Step 7, training and verifying a Multi-scale Attention Convolution Mamba Network on the Indian diabetic retinopathy image dataset in step 1 to obtain a diabetic retinopathy segmentation model, and completing diabetic retinopathy segmentation based on multi-branch and visual Mamba; The method for constructing the MSMFB model in step 4 is: The MSMFB model comprises a multi-scale dilated convolution, an SS2D in a Vmamba framework and a global-local feature fusion module, wherein the SS2D is used to capture global context information across regions, obtain overall semantic relationships using local convolution and improve the recognition ability of large-scale lesion regions or sparse lesion regions; The global-local feature fusion module is used to weight process the input features based on the corresponding weight function, and combine the weighted features to form a joint representation; and generate a global weight for adjusting the overall feature response according to the joint representation, and apply it to the initial input to obtain the final weighted feature output; The method for constructing the Joint-Weighted Skip Module in step 5 is: The input multiple feature maps are multiplied by the weights obtained by the sigmoid function, and weighted and fused to construct a feature fusion layer for fusing multi-level feature information; and the feature map of the fused multi-level feature information is output to complete the construction of the Joint-Weighted Skip Module.

2. The multi-branch and visual manhattan based diabetic retinopathy segmentation method of claim 1, wherein, The preprocessing in step 1 includes the steps of reading an image, gray-scale labeling, random rotation and flipping, random cropping to 256x256, converting a numpy data type, and adjusting a channel dimension.

3. The multi-branch and visual manhattan based diabetic retinopathy segmentation method of claim 1, wherein, The method for dividing the diabetic retinopathy images obtained in step 1 into a training set and a test set in step 2 is: adopting a five-fold cross-validation manner to process the Indian diabetic retinopathy image dataset in step 1, and the number of pictures of the training set and the test set is 54 and 27 respectively.

4. The multi-branch and visual manhattan based diabetic retinopathy segmentation method of claim 1, wherein, The method for constructing the MSAF Block model in step 3 is: The diabetic retinopathy images preprocessed in step 1 are randomly divided into sub-regions with a size of 256x256, and a multi-path attention block Multi-Path Attention Block and a dual-path cross attention fusion block Dual-Path Cross Attention Fusion Block are executed on the feature matrix of each sub-region.

5. The multi-branch and visual manhattan based diabetic retinopathy segmentation method of claim 4, wherein, The method for executing the multi-path attention block Multi-Path Attention Block operation is: the multi-path attention block Multi-Path Attention Block includes parallel attention convolution modules and multi-level down-sampling modules; wherein the attention convolution module is used to connect the coordinate attention mechanism and the residual connection; the multi-level down-sampling module is used to extract features through multi-level pooling down-sampling and a series of convolution operations, and the obtained features are up-sampled to the original spatial scale level by level.

6. The multi-branch and visual manhattan based diabetic retinopathy segmentation method of claim 4, wherein, The method for executing the dual-path cross attention fusion block Dual-Path Cross Attention Fusion Block operation on the obtained feature matrix is: the obtained multi-scale features are input into a dual-branch fusion module, and the final encoding output features are generated through a multi-scale pooling and spatial attention parallel strategy.

7. A computer storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to realize the method of any one of claims 1-6.

8. A computer device, comprising: Comprise: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to realize the method of any one of claims 1-6.

Images