CNN-Mama hybrid architecture Diffusion-based fundus image enhancement method and device

Abstract

The invention discloses a fundus image enhancement method and device based on CNN-Mama hybrid architecture Diffusion, and belongs to the field of image enhancement, and the method comprises the steps: obtaining a fundus image, and carrying out the preprocessing of the fundus image; extracting high-frequency information by using a high-frequency extractor; and combining and inputting the fundus image and the high-frequency information into the model to obtain an enhanced fundus image result. The method is based on an original Diffusion model architecture. A CNN-Mama mixed structure is designed, high-frequency and low-frequency components of an eye fundus image are separated by using a high-frequency extractor, and high-frequency information and the eye fundus image are merged and input into a model in the generation step. And a high-quality eye fundus image is provided for ophthalmology research.

Description

Technical Field

[0001] The present invention belongs to the technical field of image enhancement, and in particular relates to a fundus image enhancement method and device based on CNN-Mamba hybrid architecture Diffusion. Background Art

[0002] Retinal imaging is crucial for the diagnosis of ophthalmic diseases, but it can be affected by factors such as uneven illumination and blur, which can interfere with diagnosis. Therefore, fundus image enhancement is urgently needed. While the currently popular diffusion model boasts powerful generation capabilities and excellent detail reconstruction performance, it struggles to balance local detail with global features. Current mainstream methods mostly use CNNs or Mamba methods. CNNs focus on local details and lack the ability to extract global information, while Mamba methods focus more on global information and ignore fine edge features in the image. Summary of the Invention

[0003] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provides a fundus image enhancement method and device based on CNN-Mamba hybrid architecture Diffusion.

[0004] This paper aims to provide a hybrid architecture, called VMDiff, that fuses CNN and Mamba for fundus image enhancement, capable of simultaneously describing local details and preserving global features. Furthermore, a high-frequency extraction module and a corresponding consistent high-frequency loss are proposed to further preserve original image details, providing high-quality enhanced images for downstream tasks.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The first aspect of the present invention relates to a fundus image enhancement method based on CNN-Mamba hybrid architecture Diffusion, comprising:

[0007] S1. Acquire fundus images and preprocess the fundus images;

[0008] S2. Use a high-frequency extractor to separate the high-frequency and low-frequency components of the fundus image, amplify the structural information, and merge the high-frequency information with the fundus image into the model in the subsequent generation step; the model architecture is as follows:

[0009] H(x)=α(xg(x))

[0010] Where X is the input image, α is the coefficient for amplifying high-frequency components, which is set to 4. g(.) is a two-dimensional Gaussian kernel function with a kernel size of 16 and a standard deviation of 10.

[0011] S3. The preprocessed fundus image and high-frequency information are fed into a CNN-Mamba hybrid architecture diffusion model to obtain enhanced results. The CNN-Mamba hybrid architecture is a symmetric hybrid network structure with an encoder-decoder architecture. The encoder consists of four stages, each of which reduces the height and width of the input features while increasing the number of channels. The number of channels in each stage is [C, 2C, 3C, 4C], respectively. Similarly, the decoder is organized into four stages, each with the number of channels being [4C, 3C, 2C, C]. Finally, a convolutional block is used to restore the height and width of the features, and a projection layer is used to restore the number of channels.

[0012] Assume that the input is an image X denoted as Where H and W represent the height and width of the image respectively. Before entering the architecture, a 3×3 convolution block is first used to map the dimensions of the image patch to C, where the default size of C is 128.

[0013] X′=Conv 3×3 (X),X∈R H×W×C

[0014] The hybrid architecture mainly consists of Resnet block and VM block. Assume that the feature map input to Resnet Block at a certain stage is F in , then the output feature map F out for

[0015] F out =F in +MLP(GN(Conv(GN(F in ))))

[0016] GN is GroupNorm. VMBlock is added to the third layer of the architecture to capture global information.

[0017] The VM block module first inputs the token sequence T l-1 Normalize, then map the sequence to T′ through linear projection (LayerNorm) l-1 To the latent space of dimension E, generate features x and gate variables z:

[0018] T′ l-1 =LayerNorm(T l-1 )

[0019] x,z=Linear(T′ l-1), Linear is a linear projection layer. In the forward and backward processing, the VM Block module applies a one-dimensional convolution operation to the feature x, extracts local sequence features from it, and generates x′ o . Where SiLU is the activation function.

[0020] x′ o =SiLU(Conv1d o (x))

[0021] Then, we use linear projection to transform x′ o Convert to different weight parameters B o 、C o and the offset parameter Δ o , used to dynamically adjust sequence information.

[0022]

[0023] in, is a learnable parameter. These parameters are then fed into the state space model (SSM),

[0024]

[0025] Among them, A o is the state transition matrix, is the input projection matrix, C o is the output projection matrix, Represents a tensor multiplication operation. To calculate the forward output y forward and reverse output y backward Finally, the VM Block uses the gating variable z to adjust the fusion ratio of forward and reverse features.

[0026] y′ forward =y forward ⊙SiLU(z)

[0027] y′ backward =y backward ⊙SiLU(z)

[0028] T l ←Linear T (y′ forward +y′ backward )+T l-1

[0029] Where y′forward and y′backward are the forward and reverse features after gate modulation respectively. The final output T is generated by summing and linear operation. l .

[0030] In the Diffusion model, for a data distribution X0, X0 is high quality image, is a low-quality image, and a conditional inverse process can be learned in this distribution, which is defined as:

[0031]

[0032] in, and is from X0 and The goal of this conditional inverse process is to get the image patch from the low-quality image patch. Reconstruct the corresponding high-quality image patch At the same time, it gradually approaches the true distribution.

[0033] In order to simultaneously maintain the fidelity of low-level details and enhance high-level semantic perception, progressive loss is used as the loss function optimization strategy of VMDiff, and the overall function is recorded as

[0034] In order to optimize the learning of the conditional inverse process, the diffusion loss function is defined Its form is:

[0035]

[0036] where ∈ is the noise sampled from a standard Gaussian distribution to simulate the diffusion process, is the noise predicted by the model at step t, is a high-quality image block, is a low-quality image patch.

[0037] In order to capture more high-frequency features in the image and enhance the detailed texture of the image, a high-frequency information consistency loss function is introduced:

[0038]

[0039] in, Made of high-quality blocks The high-frequency features obtained are is made of low-quality images The obtained high-frequency features.

[0040] In order to make the generated images more consistent with human visual perception, perceptual loss is used to further train the model:

[0041]

[0042] where φ l It is the first thirty layers of the pre-trained vgg19 network, x predis the predicted image output by the model, x true is the corresponding real image. In summary, the total loss function is:

[0043]

[0044] Here, λ is 0 during the basic training phase and 50 during the fine-tuning phase.

[0045] The second aspect of the present invention relates to a fundus image enhancement device based on the CNN-Mamba hybrid architecture Diffusion, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the fundus image enhancement method based on the CNN-Mamba hybrid architecture Diffusion of the present invention.

[0046] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the fundus image enhancement method based on the CNN-Mamba hybrid architecture Diffusion of the present invention is implemented.

[0047] The innovation of the present invention is that the present invention proposes a fundus image enhancement method based on a diffusion model, which innovatively integrates the advantages of CNN and Mamba, and constructs a hybrid architecture VMDiff that takes into account both local detail modeling and global feature extraction, effectively overcoming the defect that existing methods cannot take into account both details and structures in image enhancement. At the same time, this method designs a high-frequency information extraction module and introduces a consistent high-frequency loss function, so that the enhancement results can better retain key detail features such as edges and blood vessels in the image. Experiments show that VMDiff has achieved better PSNR and SSIM performance than existing methods on multiple public fundus image datasets, and significantly improves the segmentation effect in the blood vessel segmentation task on the DRVIE dataset, verifying its practicality and wide adaptability in the field of medical image enhancement. The advantage of the present invention is that by introducing a hybrid diffusion model that integrates CNN and Mamba structures, it effectively combines the advantages of local detail enhancement and global structure modeling, and significantly improves the enhancement quality of fundus images under complex lighting and blur conditions. At the same time, the supporting high-frequency information extraction module and consistent high-frequency loss function further enhance the image detail retention ability, making the enhanced image superior to existing methods in terms of visual clarity and medical usability, with higher image quality, stronger generalization ability and wider clinical application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of the fundus image enhancement method based on the CNN-Mamba hybrid architecture Diffusion of the present invention;

[0049] Figure 2 Schematic diagram of the structure of the deep learning model of the present invention;

[0050] Figure 3 It is a schematic diagram of the device of the present invention. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0053] Example 1

[0054] This embodiment provides a fundus image enhancement method based on a CNN-Mamba hybrid architecture called Diffusion. It presents a hybrid architecture combining CNN and Mamba, called VMDiff, for fundus image enhancement, enabling the simultaneous description of local details and preservation of global features. Furthermore, a high-frequency extraction module is proposed, along with a corresponding consistent high-frequency loss, to further preserve original image details and provide high-quality enhanced images for downstream tasks.

[0055] like Figure 1 As shown,

[0056] Step 1: Acquire fundus images and preprocess the fundus images.

[0057] In order to meet the neural network input requirements, this embodiment needs to scale the remote sensing image to an image size of 512 × 512. The preprocessing in this embodiment focuses on scaling the image size. In other embodiments, other image preprocessing can be performed to improve image quality.

[0058] Step 2: Input the preprocessed fundus image into the high-frequency extractor to extract high-frequency information to obtain high-frequency information.

[0059] A high-frequency extractor is used to separate the high-frequency and low-frequency components of the fundus image, amplify the structural information, and then merge the high-frequency information with the fundus image and input it into the model in the subsequent generation step; the model architecture is as follows:

[0060] H(x)=α(xg(x))

[0061] Where X is the input image, α is the coefficient for amplifying high-frequency components, which is set to 4. g(.) is a two-dimensional Gaussian kernel function with a kernel size of 16 and a standard deviation of 10.

[0062] Step 3: Combine the fundus image and high-frequency information and input them into the model to obtain the enhanced fundus image result.

[0063] The pre-processed fundus image and high-frequency information are input into the CNN-Mamba hybrid architecture Diffusion model to obtain the enhanced result. The CNN-Mamba hybrid architecture has a symmetrical hybrid network structure and adopts an encoder-decoder structure. The encoder consists of four stages. Each stage reduces the height and width of the input features while increasing the number of channels. The number of channels in each stage is [C, 2C, 3C, 4C]. Similarly, the decoder is organized into 4 stages, and the number of channels in each stage is [4C, 3C, 2C, C]. Finally, a convolution block is used to restore the height and width of the features, and then the number of channels is restored through the projection layer. Figure 2 shown.

[0064] Assume that the input is an image X denoted as Where H and W represent the height and width of the image respectively. Before entering the architecture, a 3×3 convolution block is first used to map the dimensions of the image patch to C, where the default size of C is 128.

[0065] X′=Conv 3×3 (X),X∈R H×W×C

[0066] The hybrid architecture mainly consists of Resnet block and VM block. Assume that the feature map input to Resnet Block at a certain stage is F in , then the output feature map F out for

[0067] F out =F in +MLP(GN(Conv(GN(F in ))))

[0068] GN stands for GroupNorm, Conv stands for Convolutional Block, and MLP stands for Multi-Layer Perceptron. VMBlock is added to the third layer of the architecture to capture global information.

[0069] The VM block module first inputs the token sequence T l-1 Normalize, then map the sequence to T′ through linear projection (LayerNorm) l-1To the latent space of dimension E, generate features x and gate variables z:

[0070] T′ l-1 =LayerNorm(T l-1 )

[0071] x,z=Linear(T′ l-1 ), Linear is a linear projection layer. In the forward and backward processing, the VM Block module applies a one-dimensional convolution operation to the feature x, extracts local sequence features from it, and generates x′ o .

[0072] x′ o =SiLU(Conv1d o (x))

[0073] Then, we use linear projection to transform x′ o Convert to different weight parameters B o 、C o and the offset parameter Δ o , used to dynamically adjust sequence information.

[0074]

[0075] in, is a learnable parameter. These parameters are then fed into the state space model (SSM),

[0076]

[0077] Among them, A o is the state transition matrix, is the input projection matrix, C o is the output projection matrix, Represents a tensor multiplication operation. To calculate the forward output y forward and reverse output y backward Finally, the VM Block module uses the gate variable z to adjust the fusion ratio of forward and reverse features.

[0078] y′ forward =y forward ⊙SiLU(z)

[0079] y′ backward =u backward ⊙SiLU(z)

[0080] T l ←Linear T (y′ forward +u′ backward)+T l-1

[0081] Where y′forward and y′backward are the forward and reverse features after gate modulation respectively. The final output T is generated by summing and linear operation. l .

[0082] In the Diffusion model, for a data distribution X0, X0 is high quality image, is a low-quality image, and a conditional inverse process can be learned in this distribution, which is defined as:

[0083]

[0084] in, and is from X0 and The goal of this conditional inverse process is to get the image patch from the low-quality image patch. Reconstruct the corresponding high-quality image patch At the same time, it gradually approaches the true distribution.

[0085] In order to simultaneously maintain the fidelity of low-level details and enhance high-level semantic perception, progressive loss is used as the loss function optimization strategy of VMDiff, and the overall function is recorded as

[0086] In order to optimize the learning of the conditional inverse process, the diffusion loss function is defined Its form is:

[0087]

[0088] where ∈ is the noise sampled from a standard Gaussian distribution to simulate the diffusion process, is the noise predicted by the model at step t, is a high-quality image block, is a low-quality image patch.

[0089] In order to capture more high-frequency features in the image and enhance the detailed texture of the image, a high-frequency information consistency loss function is introduced:

[0090]

[0091] in, Made of high-quality blocks The high-frequency features obtained are is made of low-quality images The obtained high-frequency features.

[0092] This application uses batch training with a batch size of 8 (i.e., 8 images per batch). The optimizer is AdamW, with an initial learning rate of 2e-4 and a weight decay of 0.01. The noise scheduling parameter β1 and the variance scheduling parameter β2 of Diffusion are set to 0.9 and 0.999, respectively.

[0093] Example 2

[0094] like Figure 3 This embodiment relates to a fundus image enhancement device based on a CNN-Mamba hybrid architecture Diffusion, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the fundus image enhancement method based on a CNN-Mamba hybrid architecture Diffusion of Example 1.

[0095] Example 3

[0096] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the fundus image enhancement method based on the CNN-Mamba hybrid architecture Diffusion of Example 1 is implemented.

[0097] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A fundus image enhancement method based on CNN-Mamba hybrid architecture diffusion, characterized by: S1. Acquire fundus images and preprocess the fundus images; S2. Use a high-frequency extractor to separate the high-frequency and low-frequency components of the fundus image, amplify the structural information, and merge the high-frequency information with the fundus image and input it into the model in a subsequent generation step; S3. The preprocessed fundus image and high-frequency information are input into the CNN-Mamba hybrid architecture Diffusion model to obtain the enhanced results.

2. The fundus image enhancement method based on CNN-Mamba hybrid architecture Diffusion according to claim 1, characterized in that: The preprocessing described in step S1 includes scaling the fundus image to an image size of 512×512.

3. The fundus image enhancement method based on CNN-Mamba hybrid architecture diffusion as claimed in claim 1, characterized in that: It is characterized in that The model architecture of the high frequency extractor described in step S2 is: H(x)=α(xg(x)) Where X is the input image, α is the coefficient for amplifying high-frequency components, which is set to 4; g(.) is a two-dimensional Gaussian kernel function with a kernel size of 16 and a standard deviation of 10.

4. The fundus image enhancement method based on CNN-Mamba hybrid architecture Diffusion according to claim 1, characterized in that: The CNN-Mamba hybrid architecture described in step S3 is a symmetric hybrid network structure with an encoder-decoder structure. The encoder consists of four stages, each of which reduces the height and width of the input features while increasing the number of channels. The number of channels in each stage is [C, 2C, 3C, 4C]. Similarly, the decoder is organized into four stages, with the number of channels in each stage being [4C, 3C, 2C, C]. Finally, a convolution block is used to recover the height and width of the features, and then a projection layer is used to recover the number of channels. Assume that the input is an image X denoted as Where H and W represent the height and width of the image respectively; before entering the architecture, a 3×3 convolution block is first used to map the dimensions of the image block to C, where the default size of C is 128; X′=Conv 3×3 (X),X∈R H×W×C 。 5. The fundus image enhancement method based on CNN-Mamba hybrid architecture Diffusion according to claim 4, characterized in that: The hybrid architecture of the CNN-Mamba hybrid architecture described in step S3 includes a Resnet block and a VM block; assuming that the feature map input to the Resnet Block at a certain stage is F in , then the output feature map F out for: F out =F in +MLP(GN(Conv(GN(F in )))) GN stands for GroupNorm, Conv stands for Convolutional Block, and MLP stands for Multi-layer Perceptron. VMBlock is added to the third layer of the architecture to capture global information. The VM block module first inputs the token sequence T l-1 Normalize, then map the sequence to T′ through linear projection (LayerNorm) l-1 To the latent space of dimension E, generate features x and gate variables z: T′ l-1 =LayerNorm(T l-1 ) x,z=Linear(T′ l-1 ), Linear is a linear projection layer; in the forward and backward processing, the VM Block module applies a one-dimensional convolution operation to the feature x, extracts local sequence features from it, and generates x′ o , where SiLU is the activation function; x′ o =YesLU(Conv1d o (x)) Then, we use linear projection to transform x′ o Convert to different weight parameters B o 、C o and the offset parameter Δ o ,used to dynamically adjust sequence information; in, is a learnable parameter. These parameters are then fed into the state space model (SSM), Among them, A o is the state transition matrix, is the input projection matrix, C o is the output projection matrix, Represents a tensor multiplication operation; to calculate the forward output y forward and reverse output y backward ;Finally, the VM Block module uses the gate variable z to adjust the fusion ratio of forward and reverse features; and f ′ orward =and forward ⊙SiLU(z) and b ′ ackward =and backward ⊙SiLU(z) T l ←Linear T (y′ forward +y′ backward )+T l-1 Where y′forward and y′backward are the forward and reverse features after gate modulation respectively; the final output T is generated by summing and linear operation. l .

6. The fundus image enhancement method based on CNN-Mamba hybrid architecture Diffusion according to claim 1, characterized in that: The Diffusion model described in step S3, for a data distribution X0, X0 is high quality image, is a low-quality image, and a conditional inverse process is learned in this distribution, which is defined as: in, and is from X0 and The image patch is obtained by cropping; the goal of this conditional inverse process is to get the low-quality image patch Reconstruct the corresponding high-quality image patch At the same time, it gradually approaches the true distribution; In order to simultaneously maintain the fidelity of low-level details and enhance high-level semantic perception, progressive loss is used as the loss function optimization strategy of VMDiff, and the overall function is recorded as In order to optimize the learning of the conditional inverse process, the diffusion loss function is defined Its form is: where ∈ is the noise sampled from a standard Gaussian distribution to simulate the diffusion process, is the noise predicted by the model at step T, is a high-quality image block, is a low-quality image patch; In order to capture more high-frequency features in the image and enhance the detailed texture of the image, a high-frequency information consistency loss function is introduced: in, Made of high-quality blocks The high-frequency features obtained are is made of low-quality images The obtained high-frequency features; In order to make the generated images more consistent with human visual perception, perceptual loss is used to further train the model: where φ l It is the first thirty layers of the pre-trained vgg19 network, x pred is the predicted image output by the model, x true is the corresponding real image; in summary, the total loss function is: Here, λ is 0 during the basic training phase and 50 during the fine-tuning phase.

7. A fundus image enhancement device based on CNN-Mamba hybrid architecture Diffusion, characterized in that: The method comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the method is used to implement the fundus image enhancement method based on the CNN-Mamba hybrid architecture Diffusion according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the fundus image enhancement method based on the CNN-Mamba hybrid architecture Diffusion described in any one of claims 1 to 6 is implemented.

Images