Eye fundus image segmentation method based on multistage aggregation diffusion model

Abstract

The invention discloses a fundus image segmentation method based on a multistage aggregation diffusion model. The fundus image segmentation method comprises the following steps: step 1, collecting and selecting a suitable fundus image data set; 2, preprocessing the image data for model training and testing; step 3, constructing a multi-stage polymerization diffusion model MlADiff; 4, processing the output of the diffusion model encoder by using a noise suppression block; 5, extracting features in an encoder by using a variant residual block, and reducing the size of a feature map through a Downsample layer; step 6, enhancing features; step 7, feature fusion; step 8, decoding and up-sampling; step 9, processing a residual block of a pass-through body; and step 10, outputting a fundus image segmentation result, and evaluating the performance of the model by using an index. The eye fundus image segmentation accuracy and efficiency are improved.

Description

Technical Field

[0001] The invention belongs to the field of medical image segmentation and relates to a fundus image segmentation method based on a multi-level convergent diffusion model. Background Art

[0002] Fundus images are an important branch in the field of medical imaging. They are mainly used to evaluate the health of the back of the eye and are of great significance for the diagnosis and monitoring of various ophthalmic diseases such as diabetic retinopathy and macular degeneration. Fundus image segmentation refers to the identification and segmentation of specific structures in fundus images, such as blood vessels, optic discs, macula, etc., in order to provide detailed information for doctors' diagnosis and treatment. However, the imaging process of fundus images is often affected by factors such as noise, artifacts, and uneven illumination, which will reduce the quality of the image and affect the accuracy of segmentation.

[0003] With the rise of diffusion models in the field of vision, some researchers have begun to try to apply diffusion models to fundus image processing. Diffusion models can learn the probability distribution of data by simulating the diffusion process of data, generate high-quality fundus images, and help enhance and generate images. However, when processing fundus image data, the generalization ability of diffusion models may be insufficient. Fundus image data usually has complex structures and features, and the model may perform poorly on unseen data, requiring stronger generalization ability and robustness. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the present invention proposes a fundus image segmentation method based on a multi-level convergent diffusion model to improve the accuracy and efficiency of fundus image segmentation.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A fundus image segmentation method based on a multi-level convergent diffusion model comprises the following steps:

[0007] Step 1: Collect and select a suitable fundus image dataset;

[0008] Step 2: Preprocess the image data, such as resizing and enhancing quality, for model training and testing;

[0009] Step 3: Construct a multi-level aggregate diffusion model (MlADiff), which includes a diffusion model and a conditional encoder, and use skip connections and noise suppression blocks to improve model performance.

[0010] Step 4: Apply the noise suppression block (NS Block) to process the output of the diffusion model encoder and suppress high-frequency noise through FFT and IFFT;

[0011] Step 5: Use the Variant Residual Block (V Res Block) to extract features in the encoder and reduce the size of the feature map through the Downsample layer;

[0012] Step 6: The conditional encoder (Cond Encoder) processes the input image based on Resnet34, and multiplies the output features with the features processed by NSBlock to enhance the features;

[0013] Step 7: Fuse the conditional encoder and diffusion model encoder features to obtain richer feature information through the DCA module;

[0014] Step 8: The decoder restores the feature map size through skip connections and upsampling layers, integrating the low-level features of the encoder;

[0015] Step 9: Extract features in the encoder and decoder through the variant residual block (V Res Block) processed by group normalization, SiLU activation layer and convolution layer;

[0016] Step 10: Output the fundus image segmentation results and use indicators to evaluate the model performance.

[0017] Preferably, in step 10, the indicators include DSC, IoU, PRE and ACC.

[0018] The present invention proposes a multi-level aggregated network structure, called a multi-level aggregated diffusion model, which aims to fuse multi-scale features using a progressive network structure; considering that the inputs of the diffusion model are all noisy data, the present invention proposes a noise suppression block to deal with this problem. Specifically, the module realizes feature processing through fast Fourier transform and pooling technology; the features of the diffusion model encoder are processed by the noise suppression block and then passed to the conditional encoder, and the features are converted into frequency domain space through Fourier transform, thereby suppressing high-frequency noise components; such a design helps the model to make full use of local detail information and global context information; in the decoder part, the features of the decoder are combined with the features of the corresponding encoder through jump connections to obtain the final segmentation result.

[0019] The beneficial effects of the present invention are mainly manifested in that by aggregating the features of the two encoders, the model can fully grasp the local detail information and global context information, thereby improving the accuracy and efficiency of fundus image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the fundus image segmentation method based on the multi-level convergent diffusion model.

[0021] Figure 2 It is the structural diagram of the multi-level aggregation diffusion model.

[0022] Figure 3 It is a schematic diagram of the diffusion model decoder structure.

[0023] Figure 4 It is a schematic diagram of the diffusion model decoder structure.

[0024] Figure 5 It is a schematic diagram of the variant residual block structure.

[0025] Figure 6 This is a schematic diagram of the NS Block structure. DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with the accompanying drawings.

[0027] Reference Figure 1 to Figure 6 , a fundus image segmentation method based on a multi-level convergent diffusion model, comprising the following steps:

[0028] Step 1: Collect and select a 3D image dataset. Collect and select a suitable fundus image dataset.

[0029] Step 2: Preprocess the image data. Preprocess the collected image data, including adjusting the image size and enhancing the image quality, for model training and testing.

[0030] Step 3: Construct a multi-level aggregation diffusion model based on Figure 2 The internal structure of the network is shown in Figure 2, and a multi-level aggregation diffusion model (MlADiff) is constructed. Figure 2 In the paper, the whole network is divided into two parts, one is the diffusion model and the other is the conditional encoder. The diffusion model part is a classic encoder-decoder network structure. The jump connection connects the feature map in the encoder (downsampling path) with the feature map in the corresponding position in the decoder (upsampling path) to form a direct channel, so that the network can use the underlying and high-level features for image segmentation tasks at the same time. The diffusion model encoder is divided into four layers. The outputs of the first two layers of encoders are processed by the proposed noise suppression block (NS Block) and then passed to the conditional encoder. Due to the different input features of the two encoders, the input of the diffusion model encoder is the noisy label image, while the input of the conditional encoder is the fundus image. The noise suppression block transfers the noisy features of the diffusion model encoder to the features of the frequency domain interval through fast Fourier transform for processing, thereby suppressing the high-frequency noise components therein. By combining the features of the two encoders, the model can fully learn the information in the fundus image and the label, thereby obtaining a segmentation result that is closer to the label.

[0031] Step 4: Noise suppression processing: Apply a noise suppression block (NS Block) to the output of the diffusion model encoder, convert the feature map to the frequency domain through fast Fourier transform, and suppress high-frequency noise components. The structure of the noise suppression block (NS Block) is as follows: Figure 6 As shown in the figure, in the noise suppression block, the input features are first converted into frequency domain interval features using the fast Fourier transform, and then the feature distribution is adjusted through a learnable attention map. After the adjustment, it is transferred to the original feature space through the inverse Fourier transform. With the help of the fast Fourier transform, the high-frequency components in the features can be constrained to reduce the negative impact of high-frequency noise in this process. Finally, the features are multiplied with the features processed by the global average pooling layer to highlight the important information in the image. From this point of view, the noise suppression block can be regarded as a learnable frequency filter that can adaptively learn to constrain high-frequency components, thereby improving the quality of the features. The whole process is described as follows: First, given a feature map m, the frequency domain interval feature M is obtained through a 2D fast Fourier transform. Then, by parameterizing the attention map A∈C H×W×C Multiply by M1 to adjust the spectrum of m. Then use the inverse Fourier transform F -1 [·] is converted back to the original feature space to obtain M2, and finally the final output M3 is obtained through skip connection and global average pooling GAP[·]:

[0032] M = F[m]∈C H×W×C (1)

[0033]

[0034] M2 = F -1 [M1 ] (3)

[0035]

[0036] exist Figure 6 In the above figure, FFT stands for 2D Fast Fourier Transform, IFFT stands for Inverse Fourier Transform, A stands for Learnable Attention Map, GAP stands for Global Average Pooling, Stands for element-wise multiplication.

[0037] Step 5: Feature extraction and encoding: Use the variant residual block (V Res Block) to extract features in the encoder, and use the Downsample layer to downsample to reduce the size of the feature map and increase the number of channels;

[0038] Step 6. Condition Encoder (Cond Encoder) processing The conditional encoder is composed of Resnet34, which has a total of three layers. The outputs of the first two layers of encoders are multiplied by the diffusion model encoder features processed by the noise suppression block. This design enables the conditional encoder to better locate the area of ​​interest in the fundus image, thereby providing more targeted and effective features for subsequent processing. Through this mechanism, the entire model can perform image segmentation more accurately and provide more accurate and reliable results for fundus image segmentation.

[0039] Step 7: Feature fusion: pass the output features of the condition encoder (Cond Encoder) and the third layer output of the diffusion model encoder to the DCA module for feature fusion to obtain richer and more comprehensive feature information. The main part of the multi-level aggregation diffusion model is the diffusion model part, which consists of a traditional encoder-decoder structure. The encoder and decoder are implemented using a variant of ResNet. Figure 3 As shown in the figure, each diffusion model encoder (Diff Encoder) contains two variant residual blocks (V ResBlock) and a Downsample layer. The encoder mainly extracts features from the input image and reduces its size while increasing the number of channels. The features are processed by the Downsample layer, that is, downsampled by a 3×3 convolution with a stride of 2, and then used as the input of the next encoder.

[0040] The overall process of the encoding part is as follows: before the noisy labeled image enters the first encoder, it will first pass through a linear embedding layer, which will expand the channel dimension of the input image to prepare for subsequent feature extraction. Subsequently, the image processed by the linear embedding layer will be sent to the encoder, which will perform feature extraction through the variant residual block V Res Block, which helps to capture important information in the image. Subsequently, the number of channels of the image is further expanded through the downsampling layer, while the size of the feature map is reduced to reduce the size of the feature dimension during the encoding process.

[0041] Step 8, decoding and upsampling: The decoder is used to integrate and upsample the low-level features obtained by the encoder, and the size of the feature map is gradually restored to the original input image resolution through skip connections and upsampling layers. The decoder is used to integrate and upsample the low-level features obtained by the encoder. The structure of the diffusion model decoder (Diffusion ModelDncoder, Diff Dncoder) is similar to that of the encoder, such as Figure 4 As shown. represents the concat operation. The upsample layer consists of linear interpolation and a 3×3 convolution layer. The decoder receives the intermediate features E of the encoder through a skip connection. i , and then concatenate them in the channel dimension, i.e., concat operation, and then obtain the output feature D through two variant residual blocks and upsampling layers i The upsampling layer expands the size of the feature map through linear interpolation, and then reduces the number of channels through a 3×3 convolution layer to achieve the purpose of restoration. After the features are processed by layers of decoders, they can gradually restore the same resolution as the original input image.

[0042] Step 9, variant residual block processing: V Res Block is used in the encoder and decoder to process features, and important information is extracted through group normalization, activation layer and convolution layer. The internal structure of the variant residual block V Res Block is as follows Figure 5 As shown in the figure, each residual block consists of two convolution blocks, each of which consists of a group normalization GN, a SiLU activation layer, and a 3×3 convolution layer. The temporal encoding embedding t is processed by the SiLU activation layer and the linear layer to process the output of the first convolution block, and the scale is shifted so as to serve as the input of the second convolution block. In addition, a jump connection is established between the original input and the output of the second convolution block. Through the jump connection, each residual block in the diffusion model encoder can learn a residual function instead of directly learning the original mapping. This makes it easier to train very deep neural networks because jump connections can help information propagate faster in the network, alleviate the gradient vanishing problem, and also help reduce optimization difficulties in training.

[0043] Step 10: Output and performance evaluation: The final result of the decoder is output as the segmentation result of the fundus image, and the performance is evaluated using indicators such as DSC coefficient, IoU, PRE, and ACC.

[0044] This embodiment implements the proposed multi-level aggregation diffusion model network structure in the Pytorch framework. All experiments are performed on an NVIDIA GeForce GTX2080Ti graphics card with a memory capacity of 12G. In all training processes, the adaptive moment estimation optimization strategy with weight decay (Adam with Weight DecayFix, ADAMW) optimizer with a learning rate of 1e-4 is used for learning and training, and the batch size of the training is set to 1. The network model uses the Gaussian noise denoising method commonly used in the diffusion model to denoise the label image. Under normal circumstances, the model will converge stably after training 50,000 time steps.

[0045] This embodiment evaluates the proposed multi-level aggregation diffusion model network structure on multiple data sets, demonstrating the effectiveness of the multi-level aggregation diffusion model. At the same time, in order to verify the feasibility of the multi-level aggregation diffusion model, this embodiment conducts experimental comparisons on several data sets, using four indicators: DSC coefficient, IoU, PRE, and ACC for evaluation.

[0046] Since the image segmentation task in this embodiment is a binary classification, binary cross entropy is selected as the loss function. The binary cross entropy loss function is a commonly used loss function for measuring the difference between two probability distributions. In a binary classification problem, it is usually used to evaluate the difference between the predicted value output by the model and the actual label. Its mathematical expression is as follows:

[0047]

[0048] Among them, y is the prediction result of the network, is the label of the segmented image;

[0049] In this embodiment, several data sets use the same evaluation index to verify the effectiveness of the multi-level aggregation diffusion model network structure. In addition, an ablation experiment is performed on the multi-level aggregation diffusion model, which fully verifies that the method of the present invention can effectively extract the global information of the image without losing local detail information, thereby improving the accuracy of the model in the fundus image segmentation task.

[0050] The multi-level aggregation diffusion model is compared with the CNN-based networks U-Net, ResGANet, DconnNet, and some Transformer-based network architectures, including Swin-UNet and TransUNet, and is also compared with the diffusion model-based networks, including DiT, DDPM, and BerDiff.

[0051] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A fundus image segmentation method based on a multi-level convergent diffusion model, characterized in that: The method comprises the following steps: Step 1: Collect and select a suitable fundus image dataset; Step 2: Preprocess the image data for model training and testing; Step 3: Construct a multi-level aggregate diffusion model MlADiff, which includes a diffusion model and a conditional encoder, and use skip connections and noise suppression blocks to improve model performance. Step 4: Apply the noise suppression block to process the output of the diffusion model encoder to suppress high-frequency noise through FFT and IFFT; Step 5: Use the variant residual block to extract features in the encoder and reduce the size of the feature map through the Downsample layer; Step 6: The conditional encoder processes the input image based on Resnet34, and multiplies the output features with the features processed by the noise suppression block to enhance the features; Step 7: Fuse the conditional encoder and diffusion model encoder features to obtain richer feature information through the DCA module; Step 8: The decoder restores the feature map size through skip connections and upsampling layers, integrating the low-level features of the encoder; Step 9: Extract features in the encoder and decoder through the variant residual blocks processed by group normalization, SiLU activation layer and convolution layer; Step 10: Output the fundus image segmentation results and use indicators to evaluate the model performance.

2. The fundus image segmentation method based on the multi-level convergent diffusion model according to claim 1, characterized in that: In step 2, the preprocessing includes adjusting the image size and enhancing the image quality.

3. The fundus image segmentation method based on the multi-level convergent diffusion model according to claim 1 or 2, characterized in that: In the step 3, the entire network is divided into two parts, one part is the diffusion model, and the other part is the conditional encoder. The diffusion model part is a codec network structure, and the jump connection connects the encoder, that is, the feature map in the downsampling path, and the decoder, that is, the feature map at the corresponding position in the upsampling path, to form a direct channel, so that the network can simultaneously use the underlying and high-level features for image segmentation tasks; the diffusion model encoder is divided into four layers, and the outputs of the first two layers of encoders are processed by the proposed noise suppression block and then passed to the conditional encoder; due to the different input features of the two encoders, the input of the diffusion model encoder is a noisy label image, while the input of the conditional encoder is a fundus image; the noise suppression block transfers the noisy features of the diffusion model encoder to the features of the frequency domain interval through fast Fourier transform for processing, thereby suppressing the high-frequency noise components therein; by combining the features of the two encoders, the multi-level aggregate diffusion model MlADiff fully learns the information in the fundus image and the label, thereby obtaining a segmentation result that is closer to the label.

4. The fundus image segmentation method based on the multi-level convergent diffusion model according to claim 1 or 2, characterized in that: In step 4, the noise suppression block converts the feature map to the frequency domain through fast Fourier transform and suppresses the high-frequency noise component. In the noise suppression block, the input feature is first converted into a feature of the frequency domain interval using fast Fourier transform, and then its feature distribution is adjusted through a learnable attention map. After adjustment, it is transferred to the original feature space through inverse Fourier transform; finally, the features are multiplied with the features processed by the global average pooling layer to highlight the important information in the image.

5. The fundus image segmentation method based on the multi-level convergent diffusion model according to claim 4, characterized in that: The process of step 4 is as follows: first, given a feature map m, the frequency domain interval feature M is obtained by 2D fast Fourier transform; then, the parameterized attention map A∈C H×W×C Multiply by M1 to adjust the spectrum of m; then use the inverse Fourier transform F -1 [·] is converted back to the original feature space to obtain M2, and finally the final output M3 is obtained through skip connection and global average pooling GAP[·]: M = F[m]∈C H×W×C (1) M2 = F -1 [M1 ] (3) Among them, FFT stands for 2D fast Fourier transform, IFFT stands for inverse Fourier transform, A stands for learnable attention map, GAP stands for global average pooling, Stands for element-wise multiplication.

6. The fundus image segmentation method based on the multi-level convergent diffusion model according to claim 1 or 2, characterized in that: In step 6, the processing condition encoder is composed of Resnet34, which has three layers in total. The outputs of the first two layers of encoders are multiplied by the diffusion model encoder features processed by the noise suppression block.

7. The fundus image segmentation method based on the multi-level convergent diffusion model according to claim 1 or 2, characterized in that: In step 7, the output features of the conditional encoder are transmitted to the DCA module together with the third layer output of the diffusion model encoder to perform feature fusion to obtain richer and more comprehensive feature information; The main part of the multi-level aggregation diffusion model is the diffusion model, which consists of an encoder-decoder structure. The encoder and decoder are both implemented using a variant of ResNet. Each diffusion model encoder contains two variant residual blocks and a Downsample layer; the encoder extracts features from the input image and reduces its size while increasing the number of channels. The features are processed by the Downsample layer, that is, downsampled by a 3×3 convolution with a stride of 2, and then used as the input of the next encoder.

8. The fundus image segmentation method based on the multi-level convergent diffusion model according to claim 7, characterized in that: The process of the encoding part is as follows: before the noisy labeled image enters the first encoder, it will first pass through a linear embedding layer, which will expand the channel dimension of the input image to prepare for subsequent feature extraction; then, the image processed by the linear embedding layer will be sent to the encoder, and the encoder will perform feature extraction through the variant residual block V Res Block, which helps to capture important information in the image; then, the number of channels of the image will continue to be expanded through the downsampling layer, while reducing the size of the feature map to reduce the size of the feature dimension during the encoding process.

9. The fundus image segmentation method based on the multi-level convergent diffusion model according to claim 1 or 2, characterized in that: In step 8, the decoder is used to integrate and upsample the low-level features obtained by the encoder, and the size of the feature map is gradually restored to the original input image resolution through skip connections and upsampling layers; the decoder is used to integrate and upsample the low-level features obtained by the encoder, and the upsample Upsample layer is composed of linear interpolation and 3×3 convolutional layers; the decoder receives the intermediate features E of the encoder through skip connections i , and then concatenate them in the channel dimension, i.e., concat operation, and then obtain the output feature D through two variant residual blocks and upsampling layers i ; The upsampling layer expands the size of the feature map through linear interpolation, and then reduces the number of channels through a 3×3 convolution layer to achieve the purpose of restoration; after the features are processed by layers of decoders, they can gradually restore the same resolution as the original input image.

10. The fundus image segmentation method based on the multi-level convergent diffusion model according to claim 1 or 2, characterized in that: In step 9, variant residual blocks are used in the encoder and decoder to process features, and important information is extracted through group normalization, activation layers and convolution layers; in the variant residual block, each residual block is composed of two convolution blocks, and each convolution block is composed of group normalization GN, SiLU activation layer and a 3×3 convolution layer; the temporal coding embedding t processes the output of the first convolution block after being processed by the SiLU activation layer and the linear layer, and performs a scale shift on it, so as to serve as the input of the second convolution block; in addition, a jump connection is established between the original input and the output of the second convolution block, and through the jump connection, each residual block in the diffusion model encoder learns the residual function.

Images