A medical DR image enhancement algorithm based on multi-scale network

Through the medical DR image enhancement algorithm based on multi-scale networks, using the UNet network framework and multi-scale feature extraction module, the problems such as large calculation volume and noise sensitivity of the existing traditional Chinese medicine DR image enhancement algorithm are solved, and better image enhancement is achieved Effects and details are preserved.

CN118485591BActive Publication Date: 2024-11-08SHANDONG UNIV OF TECH
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410645007.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-23
Publication Date
2024-11-08
Estimated Expiration
2044-05-23

AI Technical Summary

Technical Problem

The existing medical DR image enhancement algorithm has shortcomings in problems such as large calculation volume, noise sensitivity, low local contrast, and insufficient sharpening of boundaries, resulting in artifacts and blurred boundaries in enhanced medical DR images, reducing their practicality and Clinical application value.

Method used

Using a medical DR image enhancement algorithm based on multi-scale networks, the SFP-DFE network is established through the UNet network framework, and the shallow feature pre-extraction and the DFE module are used for deep feature extraction, combining the guide filter module and the multi-scale feature attention module. Using the Laplace pyramid to connect the encoder and decoder, optimize network parameters for image enhancement.

Benefits of technology

This algorithm can better preserve multi-scale feature information of the image, reduce noise interference, improve the contrast and detail retention ability of the image, and enhance the practicality and clinical application value of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118485591B_ABST
    Figure CN118485591B_ABST
Patent Text Reader

Abstract

The present invention discloses a medical DR image enhancement algorithm based on a multi-scale network, and relates to the technical field of medical image processing. The medical DR image enhancement algorithm provided by the present invention comprises establishing an SFP-DFE network based on a UNet network framework, including an SFP module and a DFE module; using the SFP module to pre-extract features of the medical DR image to obtain a shallow feature map; introducing a guided filter module in both the encoder and the decoder of the DFE module, and filtering the feature image using a label image; embedding a PMFA module in the encoder deep network to extract multi-scale deep features; connecting the encoder and the decoder using a Laplace pyramid to reconstruct the image; and using a loss function to evaluate the SFP-DFE network and optimize network parameters. Therefore, by adopting the above-mentioned image enhancement method, multi-scale feature information can be obtained, edge detail feature information can be better retained, and the expression ability of the network can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a medical DR image enhancement algorithm based on a multi-scale network. Background Art

[0002] The purpose of medical image enhancement is to highlight certain features in the image and amplify the contrast of the area of ​​interest in the image. It mainly adopts certain corresponding image processing techniques to emphasize useful image information or suppress irrelevant information to improve the visual effect of the image.

[0003] In the field of digital image processing technology, traditional image enhancement algorithms have certain effects in natural image and medical DR image enhancement (especially in natural image enhancement), such as histogram equalization, adaptive histogram equalization, contrast-limited adaptive histogram equalization, nonlinear enhancement based on wavelet decomposition, enhancement based on Gaussian-Laplacian pyramid, etc. However, these traditional enhancement algorithms have some inherent problems. In terms of medical DR image enhancement, histogram equalization and its improved algorithms have problems such as large computational complexity, high noise sensitivity, low local contrast, and insufficient sharpness of boundaries, which lead to artifacts and blurred boundaries in enhanced medical DR images, reducing its practicality and clinical application value; the adaptive unsharp masking algorithm uses local variance as a parameter to enhance the proportion of high-frequency components, which can enhance the edges and details of the image to a certain extent, but it is also sensitive to noise, cannot effectively enhance the parts with small contrast in the image, and is prone to artifacts; the nonlinear enhancement algorithm based on wavelet decomposition is prone to serious artifacts because the parameters are difficult to determine; the improved algorithm based on Gaussian-Laplacian pyramid adopts the method of manually presetting gain factors, and the parameters are difficult to adjust, which is not smart enough.

[0004] With the development of deep learning, more and more medical image processing and reconstruction methods based on deep learning networks have been proposed. For example, Singh et al. proposed a neural network medical image fusion algorithm based on gradient decomposition, but the enhanced image has problems of noise and uneven brightness distribution. Halupka et al. proposed an optical coherence tomography (OCT) image enhancement method based on deep learning, but it is also affected by noise. Wang Dongyue combined the UNet model based on CNN with HRNet. This method is significantly better than traditional image enhancement technology in enhancing the quality of OCT images, but the enhanced image will have noise problems. Liao Shimin et al. proposed an improved low-dose CT image enhancement network based on CycleGAN, which improves the image in terms of maintaining structural details, noise and artifact suppression, but there are also problems such as GAN network is difficult to train and prone to gradient disappearance or gradient explosion. Xia Kaijian et al. proposed a low-dose CT image enhancement algorithm based on deep residual neural network. Georgescu et al. proposed a new multi-mode multi-head convolution attention module for super-resolution of CT and MRI images.

[0005] Although deep learning-based image enhancement methods have appeared in the fields of natural image processing, medical CT and OCT, due to the particularity of DR original measurement images and the difficulty in obtaining real original measurement data, there are relatively few deep learning-based medical DR image network enhancement methods. Moreover, medical DR images are quite different from ordinary images. The density of tissues and organs in various parts of the human body is different. The human body's breathing or heartbeat movement, as well as the quality differences of the equipment used to obtain medical DR images, will all cause problems such as poor contrast, mixed noise and grating shadows, and blurred edges of details in medical DR images. Therefore, there is an urgent need for an enhancement algorithm for medical DR images to provide a reference for doctors' diagnosis. Summary of the invention

[0006] The purpose of the present invention is to provide a medical DR image enhancement algorithm based on a multi-scale network, which can better obtain multi-scale feature information, retain more edge detail features, and achieve medical DR image enhancement.

[0007] To achieve the above object, the present invention provides a medical DR image enhancement algorithm based on a multi-scale network, comprising the following steps:

[0008] S1. Based on the UNet network framework, an SFP-DFE network is established to perform an enhancement algorithm on medical DR images; wherein the SFP-DFE network includes an SFP module and a DFE module;

[0009] S2, using the SFP module to pre-extract shallow features of the medical DR image to obtain a shallow feature map;

[0010] S3, taking the label image and the shallow feature image as input, using the DFE module to perform deep feature extraction, and obtaining a first side output image, a second side output image and an enhanced medical DR image;

[0011] S4. Based on the obtained first-side output image, second-side output image and enhanced medical DR image, the SFP-DFE network is evaluated and network parameters are optimized using a loss function.

[0012] Preferably, in step S2, convolution kernels of different sizes are used to extract shallow features of the medical DR image, obtain corresponding feature maps, and connect them through channels to obtain shallow feature maps. The specific expression is as follows:

[0013]

[0014] Where S(x0) represents the shallow feature map, x0 represents the medical DR image, represents channel connection, f1(f0), f3(x0), f5(x0), and f7(x0) represent the feature maps obtained by convolution with 1×1, 3×3, 5×5, and 7×7 convolution kernels, batch normalization, and LeakyReLu activation function, respectively.

[0015] Preferably, in step S3, the DFE module is established based on the UNet network framework, including an encoder and a decoder; both the encoder and the decoder introduce a guided filter module to filter the medical DR image or feature image; a multi-scale feature attention module, i.e., a PMFA module, is embedded in the encoder deep network; and the encoder and the decoder are connected through a Laplacian pyramid to reconstruct the image.

[0016] Among them, the encoder includes convolution, batch normalization, LeakyReLu activation function and downsampling operations; the decoder includes upsampling, convolution, batch normalization and LeakyReLu activation function operations.

[0017] Preferably, the guided filter module uses the label image as the guided image, filters the input image using the guided image to obtain a filtered image, and connects the filtered image with the input image through channels, and then performs convolution, batch normalization and LeakyReLu operations to obtain an output image, as follows:

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024] Where K, L, I, and I' represent the filtered image, guided image, input image, and output image in the guided filter module, respectively; f represents the convolution, batch normalization, and LeakyReLu activation function operations; i and j represent the pixel indexes; and w j represents a square window centered on the jth pixel, ε represents the regularization parameter, and a j , b j represents the linearization parameter, Indicates a in different windows j The average value of Indicates different windows b j The average value of N is the window w j The number of pixels inside μ j Respectively j The variance and mean of the guide image L in the window, Indicates w j The average value of the input image I within the window.

[0025] Preferably, the PMFA module includes a left branch, a right branch and a middle branch: the left branch includes downsampling, convolution, batch normalization, LeakyReLu activation function, upsampling and pixel-by-pixel addition; the right branch includes downsampling, convolution, batch normalization, LeakyReLu activation function and upsampling; the middle branch includes convolution, batch normalization, LeakyReLu activation function and pixel-by-pixel multiplication.

[0026] Preferably, the left branch and the right branch respectively extract features from the input feature map to obtain a left branch feature map and a right branch feature map; through the middle branch, the left branch feature map and the right branch feature map are combined to obtain a multi-scale feature map, and the specific expression is as follows:

[0027]

[0028] Where x is the input image, L l (x) represents the left branch feature map, L r (x) represents the right branch feature map, L m (x) represents the feature map of the middle branch, L z (x) represents the multi-scale feature map, D(·) represents the downsampling function, U(·) represents the upsampling function, and f s(·) represents the feature map obtained by convolution with kernels of size s×s, s=1, 3, 5, 7, batch normalization and LeakyReLu activation function.

[0029] Preferably, the Laplacian pyramid connects the encoder and the decoder to reconstruct the image, including obtaining a Laplacian pyramid feature map based on the deep features obtained by the encoder using a Laplacian pyramid-like algorithm, as follows:

[0030]

[0031] Where D k represents the feature map obtained after the kth downsampling and guided filtering module in the encoder, La k represents the kth Laplacian pyramid feature map, D' k Indicates D k The upsampled feature map, f3(·) represents the convolution, batch normalization and LeakyReLu activation function operations using a 3×3 convolution kernel, I k Represents the feature map before the input k-th downsampling.

[0032] Preferably, the obtained multi-scale feature map is passed through a decoder to gradually restore the image size, and is connected with the Laplacian pyramid feature map of the corresponding size by pixel-by-pixel addition, and through convolution, batch normalization and Sigmoid activation function, a medical DR enhanced image and a side output image are obtained.

[0033] Preferably, in step S4, the loss function is specifically expressed as follows:

[0034] L t =αL1+βL2+γL g

[0035] L g =ζL g1 +ηL g2

[0036]

[0037]

[0038]

[0039]

[0040] Where K represents the medical DR enhanced image, I represents the label image, g1 and g2 represent the first side output image and the second side output image respectively, and L t represents the loss function, L1 represents the absolute error, L2 represents the square error, L grepresents the side penalty loss, L g1 , L g2 They represent the square errors of g1 and g2 respectively, and α, β, and γ represent L1, L2, and L g The weights of L g1 , L g2 is the positive weight coefficient, m and n represent the number of rows and columns of the medical DR image, and i and j represent the index of the pixel.

[0041] Therefore, the present invention adopts the above-mentioned medical DR image enhancement algorithm based on multi-scale network, which has the following technical effects:

[0042] (1) The SPF module uses convolution kernels of different sizes to perform shallow feature pre-extraction, which can capture feature information of different receptive domains. Channel connection can also increase the information dimension, obtain global information, and provide multi-scale feature information for the DFE module.

[0043] (2) The introduction of the guided filtering module in the DFE module can use the guided image to introduce more edge information, enrich the feature information, and eliminate some image noise;

[0044] (3) The PMFA module is embedded between the encoder and decoder. The left branch uses convolution kernels of different sizes, maximum pooling, upsampling, and pixel-by-pixel addition to achieve multi-scale deep feature extraction. The right branch is used to retain the detailed features of the original medical DR image to improve the network's expression ability.

[0045] (4) The Laplacian pyramid is used to connect the encoder and decoder, which can retain multi-scale deep feature information, enhance medical DR images, and better resist noise interference.

[0046] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the SFP-DFE network structure in an embodiment of a medical DR image enhancement algorithm based on a multi-scale network;

[0048] Figure 2 It is a schematic diagram of the structure of a guided filtering module in an embodiment of a medical DR image enhancement algorithm based on a multi-scale network;

[0049] Figure 3 It is a schematic diagram of the PMFA module structure in an embodiment of a medical DR image enhancement algorithm based on a multi-scale network;

[0050] Figure 4 It is an experimental flow chart of an embodiment of a medical DR image enhancement algorithm based on a multi-scale network;

[0051] Figure 5 is a training error curve of an SFP-DFE network in an embodiment of a medical DR image enhancement algorithm based on a multi-scale network;

[0052] Figure 6 is a cross-validation error curve of the SFP-DFE network in an embodiment of a medical DR image enhancement algorithm based on a multi-scale network;

[0053] Figure 7 A medical DR enhanced image of different network models in an embodiment of a medical DR image enhancement algorithm based on a multi-scale network;

[0054] Figure 8 It is an image enhancement result of different human body parts outside the training set in an embodiment of a medical DR image enhancement algorithm based on a multi-scale network;

[0055] Fig. 9 It is an image enhancement result of different models in ablation experiment in an embodiment of a medical DR image enhancement algorithm based on a multi-scale network. DETAILED DESCRIPTION

[0056] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0057] like Figure 1 As shown, the present invention provides a medical DR image enhancement algorithm based on a multi-scale network, comprising:

[0058] S1. Establish an SFP-DFE network based on the UNet network framework, including a shallow feature pre-extraction module (SFP module) and a deep feature extraction module (DFE module).

[0059] S2. Use the SFP module to pre-extract shallow features of the medical DR image to obtain a shallow feature map.

[0060] Based on the SFP module, convolution kernels of different sizes are used to perform convolution, BN normalization and LeakyReLu function activation, obtain the corresponding feature maps and connect them in series, i.e., channel connection (Concatenation), to obtain shallow feature information.

[0061] The sizes of the convolution kernels are 1×1, 3×3, 5×5, and 7×7, respectively, and each scale has 16 channels.

[0062] The specific expression of SFP module is as follows:

[0063]

[0064] Where S(x) represents the shallow feature image, represents channel connection, f1(x), f3(x), f5(x), and f7(x) represent the feature maps obtained by the SFP module using convolution kernels of 1×1, 3×3, 5×5, and 7×7 sizes, batch normalization (BN), and LeakyReLu function activation operations, respectively.

[0065] S3. In the DFE module, the label image (high-quality medical DR image) and the shallow feature map are used as input, the label image is selected as the guide image, and deep features are extracted from the shallow feature map to obtain a medical DR enhanced image.

[0066] Among them, the DFE module is built based on the UNet network framework, including an encoder and a decoder.

[0067] In the encoder, convolution, batch normalization (BN), LeakyReLu activation function and maximum pooling downsampling operations are performed on the shallow feature map to gradually reduce the feature image size and increase the number of channels by multiples to extract abstract features.

[0068] In addition, the guided filter module is introduced in both the encoder and the decoder, and the guided image is used to filter the feature image to obtain the filtered image. The filtered image is connected to the original image through channels, and convolution, batch normalization (BN) and LeakyReLu function activation operations are performed to obtain the output image I', as shown in Figure 2 The specific expression is as follows:

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075] Where K, L, I, and I' represent the filtered image, guided image, input image, and output image in the guided filter module, respectively; f represents the convolution, batch normalization, and LeakyReLu activation function operations; i and j represent the pixel indexes; and w j represents a square window centered on the jth pixel, ε represents the regularization parameter, and a j , b j represents the linearization parameter, Indicates a in different windows j The average value of Indicates different windows b j The average value of N is the window w j The number of pixels inside μ j Respectively j The variance and mean of the guide image L in the window, Indicates w j The average value of the input image I within the window.

[0076] In the SFP-DFE network, the guided filtering module filters the feature map obtained by downsampling the encoder and the feature map obtained by upsampling the decoder.

[0077] A multi-scale feature attention module (PMFA module) is embedded between the encoder and the decoder, such as Figure 3 As shown in the figure. The PMFA module consists of three parts: the left branch, the middle branch, and the right branch. The module performs multi-scale deep feature extraction in the feature map of the encoder's third downsampling. The left branch includes downsampling, convolution, batch normalization, LeakyReLu activation function, upsampling, and pixel-by-pixel addition; the right branch includes downsampling, convolution, batch normalization, LeakyReLu activation function, and upsampling; the middle branch includes convolution, batch normalization, LeakyReLu activation function, and pixel-by-pixel multiplication.

[0078] Among them, the left branch and the right branch extract features from the input feature map respectively to obtain the left branch feature map and the right branch feature map; through the middle branch, the left branch feature map and the right branch feature map are combined to obtain a multi-scale feature map. The specific expression is as follows:

[0079]

[0080] Where x is the input image, L l (x) represents the left branch feature map, L r (x) represents the right branch feature map, L m (x) represents the feature map of the middle branch, L z (x) represents the multi-scale feature map, D(·) represents the downsampling function, U(·) represents the upsampling function, and f s (·) represents the feature map obtained by convolution with kernels of size s×s, s=1, 3, 5, 7, batch normalization and LeakyReLu activation function.

[0081] The left branch is used to extract deep features of different scales, and the right branch is used to maintain the shallow features of the medical DR image, which are fused through the middle branch. The entire PMFA module adopts the LeakyreLu activation function to retain the negative information of the medical DR image, integrate the multi-scale feature information more accurately, and retain the details to the maximum extent, thereby improving the performance of the SMFA module.

[0082] In addition, the encoder and decoder are connected through the Laplacian pyramid to reconstruct the image, which includes the deep features obtained by the encoder and the Laplacian pyramid algorithm to obtain the Laplacian pyramid feature map, as follows:

[0083] Input: Feature map I before the kth downsampling k , and its feature map D after downsampling and processing by the guided filter module k ,k=1,2,3;

[0084] 1) Perform upsampling and one convolution + BN + LeakyReLu operation on D1 to obtain D'1;

[0085] 2) Subtract the input feature map I1 from D'1 to get La1;

[0086] 3) Perform upsampling and one convolution + BN + LeakyReLu operation on D2 to obtain D'2;

[0087] 4) Subtract the input feature map I2 from D'2 to get La2

[0088] 5) Perform upsampling and one convolution + BN + LeakyReLu operation on D3 to obtain D'3;

[0089] 6) Subtract the input feature map I3 from D'3 to get La3

[0090] 7) Return the Laplacian pyramid feature map La k ,k=1,2,3.

[0091] The specific expression is as follows:

[0092]

[0093] Where D k represents the feature map obtained after the kth downsampling and guided filtering module in the encoder, La k represents the kth Laplacian pyramid feature map, D' k Indicates D k The upsampled feature map, f3(·) represents the convolution, batch normalization and LeakyReLu activation function operations using a 3×3 convolution kernel, I kRepresents the feature map before the input k-th downsampling.

[0094] The decoder is used to gradually restore the image size of the multi-scale feature map obtained by the PMFA module, and connect it with the Laplacian pyramid feature map of the corresponding size through pixel-wise addition (Element-Wise Summation). Through convolution, batch normalization and Sigmoid activation function, the first side output image, the second side output image and the medical DR enhanced image are obtained.

[0095] S4. Based on the obtained first side output image, second side output image and medical DR enhanced image, the SFP-DFE network is evaluated by a loss function.

[0096] The loss function is specifically expressed as follows:

[0097] L t =αL1+βL2+γL g

[0098] L g =ζL g1 +ηL g2

[0099]

[0100]

[0101]

[0102]

[0103] Where L represents the medical DR enhanced image, I represents the label image, g1 and g2 represent the first side output image and the second side output image respectively, and L t represents the loss function, L1 represents the absolute error, L2 represents the square error, L g represents the side penalty loss, L g1 , L g2 They represent the square errors of g1 and g2 respectively, and α, β, and γ represent L1, L2, and L g The weights of L g1 , L g2 The positive weight coefficient of , m, n represent the number of rows and columns of the medical DR image, and i, j represent the index of the pixel.

[0104] In this embodiment, α, β, and γ are set to 0.1, 0.4, and 0.5 respectively, and ζ and η are set to 0.6 and 0.4 respectively, so that γ·ζ+γ·η=0.5, α+β=0.5, ensuring that the features of the side output layer are as important as the features of the final output layer. In addition, since the L2 loss is smaller than the L1 loss, β is selected to be larger than α.

[0105] Example Experimental Verification

[0106] according to Figure 4 The flowchart shown verifies the effect of SFP-DFE network on medical DR image enhancement, and conducts comparative experiments with existing models AHE, CLAHE, GLP, LLCNN, UNet and StillGAN.

[0107] In the comparative experiments, in order to increase the reliability of the comparative experiments, the above models all used preprocessed medical DR images that had undergone grid shadow removal, data clipping, flipping, and expansion as training data sets to increase the diversity of samples and improve the generalization and robustness of the models. In addition, cross-validation was added to the training, that is, 70% of the training set was used for model training and 30% of the training set was used for model verification, in order to evaluate the model more reasonably and accurately and avoid overfitting.

[0108] The SFP-DFE network training error curve used in the embodiment is as follows: Figure 5 As shown, the cross validation error curve is Figure 6 shown.

[0109] like Figure 7As shown, the red boxes mark the key areas in the original medical DR image and the label image, and the orange and green boxes mark the corresponding areas in the medical DR enhanced image. The results show that the SFP-DFE network (Ours) combines shallow features and deep features with the local feature attention module, which can better preserve global and local features, and its image enhancement effect is the best among all the compared methods. The images enhanced by the AHE and CLAHE algorithms are usually too bright, especially in the orange and green frame areas, with problems such as poor contrast, blurred boundaries and loss of details; the GLP algorithm uses manual adjustment factors for exponential enhancement, which improves some detail enhancement, but the overall image is still bright and the contrast is relatively low; LLCNN based on convolution operations does not require manual parameter adjustment, but the enhanced images are usually dark, especially in the parts with smaller grayscale values, which may be caused by mislearning small values ​​as background during training; the enhanced images of the LLCNN method show information loss in the fingertips and muscles, and the contrast of edge details is poor; UNet is based on encoders and decoders, and can extract and integrate image features without manual parameter adjustment, and has higher contrast, clearer edges and more details than traditional algorithms; StillGAN extracts image features based on generative adversarial networks without manual adjustment, but the enhanced images are darker, too much internal details are lost, and they are too smooth.

[0110] In addition, to further verify the effectiveness of the SFP-DFE network, the labeled images were used as references, and the evaluation indicators peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and contrast-to-noise ratio (CNR) were used to evaluate the reliability of the above network model, and a comparative analysis was performed.

[0111] Among them, PSNR is used to measure the difference between the enhanced image and the marked image. Generally, the smaller the MSE, the larger the PSNR, which means the better the quality of the enhanced image.

[0112] The specific expression of PSNR is:

[0113]

[0114]

[0115] In the formula, x(i,j) and y(i,j) represent the label image and medical DR enhanced image respectively. It represents the square of the maximum gray value in the label image L(i,j), MSE represents the mean square error, m and n represent the number of rows and columns of the medical DR image, and i and j represent the index of the pixel.

[0116] SSIM is used to measure the similarity between two images. It mainly analyzes images from three different aspects: image brightness, contrast, and structure. The specific expression is:

[0117]

[0118] In the formula, μ x , μ y Respectively represent the average values ​​of x image and y image, σ x , σ y Respectively represent the variance of x image and y image, σ x,y Represents the covariance of the x image and the y image. C1 and C2 are constants to avoid the denominator being zero. The SSIM value is in the range of [0, 1]. The closer it is to 1, the better the enhanced image quality.

[0119] CNR is mainly used to measure the ratio between the contrast of useful information and the relative intensity of noise. It is usually used to quantify the contrast and noise level of enhanced images. Generally speaking, the larger the CNR value, the higher the image contrast, the lower the noise level, and the better the image quality. The specific expression of CNR is:

[0120]

[0121]

[0122]

[0123] In the formula, μ s Represents the average gray value of useful information in the image, μ n Represents the average gray value of the noise in the image, Represents the variance of useful information in the image, represents the variance of the noise in the image, I s Represents the useful information in the image, I n Represents the noise in the image.

[0124] The evaluation indicators of different models are shown in Table 1.

[0125] Table 1 Evaluation indicators of different models

[0126]

[0127] The results in Table 1 show that the CNR of the network-based methods (LLCNN, UNet, StillGAN and SFP-DFE) is significantly improved compared with the traditional methods (AHE, CLAHE and GLP), which indicates that the network-based methods have better effects in denoising and contrast improvement. In addition, the SFP-DFE network used in the embodiment has the highest PSNR, SSIM and CNR values ​​compared with other models, which indicates that the SFP-DFE network has great potential in medical DR image enhancement.

[0128] In addition, to verify the generalization of the SFP-DFE network, we tested it on original images of human body parts that were not included in the training data. We found that the network also performed well in enhancing DR images of more complex areas, such as the thorax and lumbar spine. Figure 8 shown.

[0129] Ablation experiment

[0130] In order to verify the effectiveness of adding BN module, PMFA module, SFP module, guided filter module, LeakyReLu activation function, Laplacian pyramid connection, and loss function, an ablation experiment was conducted, and the model design is shown in Table 2. The evaluation indicators of different models in the ablation experiment are shown in Table 3.

[0131] Table 2 Description of the model

[0132] Model Model Description Purpose Ours SFP-DFE Network Study the impact of all modules in the model Model 1 Remove the BN module Study the impact of BN operation Model 2 Remove PMFA module Studying the impact of the PMFA module Model 3 Remove the SFP module Study the impact of SFP modules Model 4 Remove the guide filter module Studying the impact of guided filter modules Model 5 Use ReLu activation function Study the impact of the LeakyReLu activation function Model 6 The PMFA module is added to the encoder's first downsampling position The influence of PMFA module on the shallow layer of encoder Model 7 The PMFA module is added to the encoder's second downsampling position The influence of PMFA module in the deep encoder Model 8 The encoder and decoder are connected by element-by-element addition Studying the impact of pyramid connections Model 9 <![CDATA[Update the parameters using the L1 and L2 loss functions]]> <![CDATA[Study the influence of L1 and L2 loss functions]]> Model 10 <![CDATA[Use L s oss function to update parameters]]> <![CDATA[Study L s Impact of the loss function]]>

[0133] like Fig. 9 As shown in the figure, the image enhancement effect of model 1 is poor, which shows that the role of BN in the SFP-DFE network is very important. The image enhanced by model 2 has obvious detail loss and many white parts, which reflects that the PMFA module can play a role in detail protection; the image enhanced by model 3 has obvious detail loss, blurred edges and low contrast; the enhanced image of model 4 has more overly bright areas around the edges of bones; therefore, it shows the importance of PMFA module, SFP module and guided filter module in detail preservation and contrast improvement.

[0134] Table 3 Evaluation indicators of different models in ablation experiments

[0135]

[0136] The image contrast of model 5 is improved, and the details and boundaries are well preserved, however, there are still some small values ​​of trabecular details lost, which shows that the LeakyReLu activation function plays an important role in maintaining the negative information of the image.

[0137] The enhanced image of model 6 is too smooth, resulting in the loss of many details, indicating that the PMFA module cannot fully retain the image details in the first layer of the encoder; the enhanced image of model 7 is improved in details, but there is still some noise; therefore, the PMFA module should be added to the deepest position of the encoder.

[0138] The details and contrast preservation of the images enhanced by model 8 are significantly improved, but still not better than our method, which shows that the Laplacian pyramid connection can enhance the edge information of the image and preserve the detail information.

[0139] Model 9 combines the loss functions of L1 and L2 to slightly enhance the enhanced image, but it still suffers from edge blur, dark images, and detail loss.

[0140] Model 10 improves the contrast and details better than Model 9, which reflects that the side output penalty loss function plays an important role in preserving details and edges.

[0141] In summary, the structure, details and contrast of the SFP-DFE enhanced image are better than those of models 1-10, and can solve problems such as detail loss, edge blur and brightness. This fully demonstrates that BN, PMFA module, SFP module, guided filtering module, LeakyReLu activation function and Laplacian pyramid can obtain better multi-scale information and prevent detail loss.

[0142] Therefore, the present invention adopts the above-mentioned medical DR image enhancement algorithm based on multi-scale network, which can obtain better multi-scale information, prevent detail loss, and improve the expression ability of the network.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A medical DR image enhancement algorithm based on a multi-scale network, characterized in that: The following steps are involved: S1. Establish an SFP-DFE network based on the UNet network framework, including SFP modules and DFE modules; S2. Use the SFP module to pre-extract shallow features of medical DR images, use convolution kernels of different sizes to extract shallow features of medical DR images, obtain corresponding feature maps, and connect them through channels to obtain shallow feature maps. The specific expression is as follows: Where S(x0) represents the shallow feature map, x0 represents the medical DR image, represents channel connection, f1(x0), f3(x0), f5(x0), and f7(x0) represent feature maps obtained by convolution with 1×1, 3×3, 5×5, and 7×7 convolution kernels, batch normalization, and LeakyReLu activation function respectively; S3, taking the label image and the shallow feature image as input, using the DFE module to perform deep feature extraction, and obtaining a first side output image, a second side output image and an enhanced medical DR image; Among them, the DFE module is established based on the UNet network framework, including an encoder and a decoder; both the encoder and the decoder introduce a guided filter module to filter the medical DR image or feature image; a multi-scale feature attention module, namely, a PMFA module, is embedded in the deep network of the encoder; and the encoder and the decoder are connected through the Laplacian pyramid to reconstruct the image, including the deep features obtained by the encoder, and the Laplacian pyramid algorithm is used to obtain the Laplacian pyramid feature map, as follows: Where D k represents the feature map after the kth downsampling and guided filtering module in the encoder, La k represents the kth Laplacian pyramid feature map, D′ k Indicates D k The upsampled feature map, f3(·) represents the convolution, batch normalization and LeakyReLu activation function operations using a 3×3 convolution kernel, I k Represents the feature map before the input k-th downsampling; The encoder includes convolution, batch normalization, LeakyReLu activation function and downsampling operations; the decoder includes upsampling, convolution, batch normalization and LeakyReLu activation function operations; S4. Based on the obtained first-side output image, second-side output image and enhanced medical DR image, the SFP-DFE network is evaluated and network parameters are optimized using a loss function.

2. The medical DR image enhancement algorithm based on a multi-scale network according to claim 1, characterized in that: The guided filter module uses the label image as the guide image, filters the input image using the guide image, obtains the filtered image, and connects the filtered image with the input image through channels, and then obtains the output image through convolution, batch normalization and LeakyReLu operations, as follows: Where K, L, I, and I′ represent the filtered image, guided image, input image, and output image in the guided filter module, respectively; f represents the convolution, batch normalization, and LeakyReLu activation function operations; i and j represent the pixel indexes; and w j represents a square window centered on the jth pixel, ε represents the regularization parameter, and a j , b j represents the linearization parameter, Indicates a in different windows j The average value of Indicates different windows b j The average value of N is the window w j The number of pixels inside μ j Respectively j The variance and mean of the guide image L in the window, Indicates w j The average value of the input image I within the window.

3. The medical DR image enhancement algorithm based on a multi-scale network according to claim 2, characterized in that: The PMFA module includes a left branch, a right branch, and a middle branch: the left branch includes downsampling, convolution, batch normalization, LeakyReLu activation function, upsampling, and pixel-by-pixel addition; The right branch includes downsampling, convolution, batch normalization, LeakyReLu activation function and upsampling; the middle branch includes convolution, batch normalization, LeakyReLu activation function and pixel-by-pixel multiplication.

4. The medical DR image enhancement algorithm based on a multi-scale network according to claim 3, characterized in that: The left branch and the right branch extract features from the input feature map respectively to obtain the left branch feature map and the right branch feature map; through the middle branch, the left branch feature map and the right branch feature map are combined to obtain a multi-scale feature map. The specific expression is as follows: Where x is the input image, L l (x) represents the left branch feature map, L r (x) represents the right branch feature map, L m (x) represents the feature map of the middle branch, L z (x) represents the multi-scale feature map, D(·) represents the downsampling function, U(·) represents the upsampling function, and f s (·) indicates convolution, batch normalization and LeakyReLu activation function operations using convolution kernels of size s×s, s=1, 3, 5, 7.

5. The medical DR image enhancement algorithm based on multi-scale network according to claim 1, characterized in that: The obtained multi-scale feature map is gradually restored to the image size through the decoder, and connected with the Laplacian pyramid feature map of the corresponding size through pixel-by-pixel addition. Through convolution, batch normalization and Sigmoid activation function, the first side output image, the second side output image and the medical DR enhanced image are obtained.

6. The medical DR image enhancement algorithm based on multi-scale network according to claim 1, characterized in that: In step S4, the loss function is specifically expressed as follows: L t =αL1+βL2+γL g L g =ζL g1 +ηL g2 Where K represents the medical DR enhanced image, I represents the label image, g1 and g2 represent the first side output image and the second side output image respectively, and L t represents the loss function, L1 represents the absolute error, L2 represents the square error, L g represents the side penalty loss, L g1 , L g2 They represent the square errors of g1 and g2 respectively, and α, β, and γ represent L1, L2, and L g The weights of L g1 , L g2 The positive weight coefficient of , m, n represent the number of rows and columns of the medical DR image, and i, j represent the index of the pixel.

Citation Information

Patent Citations

  • Image fusion method based on complex shear wave transformation and deep convolution neural network

    CN108629757A

  • Deep learning-based denoising and super-resolution combined low-illumination image enhancement method

    CN116051428A

  • Multi-level lesion detection optimized pathological section image segmentation detection method

    CN116596952A

  • Retinal vessel segmentation method based on global feature enhancement

    CN116630345A

  • Mine image super-resolution reconstruction method and system based on hierarchical feature fusion

    CN117114994A