Image magnification method based on dilated convolution fusion
Through the combination of multi-level multi-scale residual structure neural network and deep separable cavity convolution, the problem of difficulty in extracting high-frequency and low-frequency information in fundus medical images is solved, and higher quality image reconstruction is achieved. And improve model performance through frequency constraint loss function.
Patent Information
- Application Number
- CN202410902633.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-12-03
AI Technical Summary
Traditional super-resolution methods are difficult to effectively extract high-frequency and low-frequency information in fundus medical image reconstruction, resulting in poor image reconstruction quality, large parameters and high computational complexity, making it easy to have gradient explosion and disappearance problems.
Using a multi-level multi-scale residual structure neural network and deep separable cavity convolution fusion, combining a dual-path residual network and a hollow convolution block, a medical fundus image super-resolution reconstruction network is designed to expand the receptive field through a hollow convolution block , extract rich image information features, and introduce frequency constraint loss functions to accelerate model convergence and improve performance.
Without increasing the parameter quantity and resolution, the image quality indicators (PSNR, SSIM) are significantly improved, the reconstruction effect of high-frequency and low-frequency details is improved, the problem of loss of high-frequency and low-frequency details in traditional methods is solved, and the problem of loss of high-frequency and low-frequency details is improved. The running speed and performance of the model.
Smart Images

Figure CN119067847A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision and deep learning, and belongs to the direction of super-resolution reconstruction. It specifically designs a method based on the fusion of a multi-scale residual structure neural network and a deep separable hole convolution. Background Art
[0002] With the change of population structure, the aging problem has become extremely serious, so the prevention of diseases has become more urgent. The sudden onset of physiological diseases may cause changes in fundus images, so fundus medical image examination is one of the important means to prevent the deterioration of physiological diseases. However, most high-precision fundus imaging devices are inconvenient to carry and expensive. In emergency situations, they can only rely on portable devices, and the image quality presented by these devices is worrying. Without increasing the economic burden, we can improve the imaging effect and resolution of these images through super-resolution technology.
[0003] Image super-resolution reconstruction is the process of converting one or more low-resolution images into corresponding high-resolution images. Although traditional interpolation methods can reconstruct high-resolution images simply and quickly, they often fail to achieve ideal results in terms of performance indicators and natural perception.
[0004] Super-resolution methods based on deep learning have achieved remarkable results on natural image datasets. Dong et al. first proposed the SRCNN super-resolution neural network based on deep learning, and later applied it to the super-resolution task of medical images. However, VDSR did not achieve significant performance indicators by using a large number of parameters and deep network structures, and often caused problems such as gradient explosion and vanishing. IMDN uses multiple information distillation modules and introduces a channel attention mechanism to more effectively utilize deep image information features, improving the network's operating speed and performance indicators without significantly increasing the number of parameters and network depth.
[0005] Eye images contain rich high-frequency information in the highlighted parts of the optic disc area and capillaries, while the dark parts of the retinal fundus images contain more low-frequency information. The application of traditional super-resolution methods on natural images often finds it difficult to ensure good image restoration effects while using fewer parameters. The method proposed in this study combines a multi-scale residual network and a dilated separable convolution, achieving higher image quality indicators (PSNR, SSIM) while using fewer parameters, which is significantly better than most natural image networks. Summary of the invention
[0006] The purpose of the present invention is to overcome the shortcomings of the prior art, and to better process the high-frequency and low-frequency image features in medical images based on the super-resolution medical image reconstruction technology of multi-level multi-scale residual network and dilated convolution, combined with dual-path residual network and dilated convolution block; the introduction of dilated convolution block increases the receptive field without increasing the number of parameters and reducing the resolution, thereby capturing richer image information features. A medical fundus image super-resolution reconstruction network is designed to improve the quality and details of medical fundus image super-resolution reconstruction.
[0007] In order to achieve the above object, the present invention adopts the following technical solution:
[0008] The invention discloses a method for fusing a multi-level multi-scale residual structure neural network with a deep separable dilated convolution. The model consists of three parts: a shallow image feature extraction network, a deep image feature extraction network, and an upsampling image reconstruction network.
[0009] The shallow feature extraction module is used to extract shallow features of medical fundus image information. It uses a 3×3 convolutional layer to extract features from low-resolution text images, maps RGB channel information into feature space, and transmits this information to the deep image feature extraction module. This feature extraction module only extracts initial features from low-resolution images to obtain shallow features of medical fundus images.
[0010] The deep image feature extraction module is used to extract the deep features of the medical fundus image. The low-resolution medical fundus image features extracted in the previous step are input into the deep feature extraction. The module includes 6 DSMM residual blocks. Each DSMM residual block uses two upper and lower paths and different numbers of DCBK hole convolution blocks and LKA large kernel convolution attention blocks to extract the deep-level fundus medical image texture, and uses residual connections to prevent gradient disappearance.
[0011] The upsampling module inputs the features extracted from the deep image features into the upsampling layer, amplifies the image by 4 times, and obtains a high-resolution medical fundus image. After passing through a 3×3 convolution block, the image is finally reconstructed and the final super-resolution medical fundus image is output;
[0012] Preferably, the DSMM residual module comprises:
[0013] This module adopts a multi-scale cross network structure, in which one path is dedicated to extracting low-frequency information of the image, and the other path is responsible for extracting high-frequency information. The high-frequency branch of DSMM consists of two dilated convolution blocks DCBK, while the low-frequency information branch consists of a single dilated convolution block. Finally, this fused information is input into the large-core attention block LKA. LKA combines three parts: local spatial convolution, spatial long-range convolution, and channel convolution, and can achieve a 21×21 receptive field. It not only retains the advantages of large-core convolution, but also overcomes the computational complexity and memory capacity requirements of the self-attention mechanism, and extracts the deep information of the image more delicately, thereby showing excellent performance in visual tasks.
[0014] Preferably, the loss function includes:
[0015] (1) Mean square error loss
[0016] The mean square error loss, also known as L1 loss, is one of the most common loss functions. It compares the difference between the generated high-resolution image and the real high-resolution image by calculating the average of the squared differences between the pixels:
[0017]
[0018] Where m is the number of samples, y i is the true value of the i-th sample, is the predicted value of the ith sample.
[0019] (2) The overall loss function can be expressed as follows:
[0020] In medical image reconstruction, since single pixel loss makes it difficult to effectively reconstruct high-frequency information, we introduce frequency constraints to regulate network training to achieve better reconstruction results. The total loss function of this algorithm includes a frequency constraint term, and its formula is as follows:
[0021]
[0022] in represents Fourier transform, λ is the weight, usually defaulted to 0.1. Introducing a frequency-constrained loss function can accelerate the convergence of the model and make the spectrum of the generated image closer to the spectrum of the real image.
[0023] The medical fundus super-resolution reconstruction method based on a multi-level multi-scale residual network includes the following steps:
[0024] S1. Download the fundus dataset STARE for capillary segmentation algorithm. During training, the dataset contains 397 retinal images with a resolution of 700×605 pixels. Since the image resolutions in the dataset are not all multiples of 4, the images need to be preprocessed first. We use MATLAB to resize the images to 696×600 so that they can be used as ground-truth images. Next, we downsample the images to generate images with a resolution of 4 times smaller and ensure that they are consistent with the original images. Figure 1 One-to-one correspondence. The experiment selected the first 100 images as the training set and the remaining 297 images as the test set;
[0025] S2. Input the processed image obtained in S1 into the shallow feature extraction module to extract the shallow features of the medical fundus super-resolution image, and use the convolution layer to extract features of the low-resolution medical fundus image. This convolution layer can only extract the shallow features of the medical fundus image to obtain the low-resolution image I. SR ;
[0026] S3. Input the low-resolution image obtained in S2 into the deep feature extraction module, and input the extracted shallow features of the low-resolution medical fundus image into the DSMM residual module for deep feature extraction. The module includes 6 residual blocks. This module adopts a multi-scale cross network structure, in which one path is specifically used to extract the low-frequency information of the image, and the other path is responsible for the extraction of high-frequency information. The high-frequency branch of DSMM consists of two dilated convolution blocks DCBK, while the low-frequency information branch consists of a single dilated convolution block. Finally, this fused information is input into the large core attention block LKA;
[0027] S4. Input the deep image features obtained in S3 into the upsampling network, input the features extracted from the deep image features into the upsampling layer, magnify the image by 4 times, and obtain a high-resolution medical fundus image. Then, after a 3×3 convolution block, the image is reconstructed and the final super-resolution medical fundus image is output;
[0028] S5. Input the high-resolution image generated in S4 into the 9×9 convolution kernel, perform the final reconstruction on the image, and output the final super-resolution image I SR ;
[0029] S6. Set the hyperparameters of the medical fundus image super-resolution network model, and set the batch size to 16. The optimizer uses the Adam algorithm, where β1=0.9, β2=0.99. The initial learning rate is set to 0.002, the minimum learning rate is 0.0000001, and the cosine annealing strategy is used to update the learning rate. The input image is cropped into small blocks of 32×32 resolution, and then image enhancement operations such as flipping are performed. Finally, the corresponding high-resolution image is output through the network for back propagation. The experiment uses PSNR and SSIM values as objective evaluation criteria.
[0030] It can be seen from the above technical solution that, compared with the prior art, the beneficial effects of the present invention are:
[0031] (1) The DCBN algorithm proposed in this paper is based on dilated separable convolution and multi-scale neural network, which solves the problem of high-frequency and low-frequency details loss in fundus medical image reconstruction by traditional super-resolution methods. Dilated convolution expands the receptive field without increasing parameters to extract information more efficiently, while multi-scale neural network can more effectively extract high-frequency and low-frequency information of images. Separable convolution can alleviate the grid effect caused by dilated convolution.
[0032] (2) The input constraint frequency can not only accelerate the convergence of the model, but also improve the model performance. The LKA large-core attention block obtains a larger receptive field with fewer parameters, thereby improving the final performance. After experimental verification, the algorithm proposed in this paper has achieved significant improvement in performance indicators compared with the traditional lightweight super-resolution reconstruction algorithm, and has improved the perception of high-frequency and low-frequency details in visual effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0034] Figure 1 is a summary drawing of the specification of the present invention;
[0035] Figure 2 It is a schematic diagram of the overall structure of a medical fundus super-resolution reconstruction method based on a multi-level multi-scale residual network of the present invention;
[0036] Figure 3 The present invention discloses a medical fundus super-resolution reconstruction method based on a multi-level multi-scale residual network using a DSMM module overall structure;
[0037] Figure 4 The present invention discloses a medical fundus super-resolution reconstruction method based on a multi-level multi-scale residual network using a DCBK convolution block;
[0038] Figure 5 The present invention discloses a medical fundus super-resolution reconstruction method based on a multi-level multi-scale residual network, in which the dataset image im330 is downsampled by 4 times, the reconstruction effect of each algorithm, and the high-frequency information part;
[0039] Figure 6 It is the reconstruction effect of each algorithm and the low-frequency information part of the dataset image im130 in a medical fundus super-resolution reconstruction method based on a multi-level multi-scale residual network of the present invention after 4 times downsampling; DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] The present invention provides a medical fundus super-resolution reconstruction method based on a multi-level multi-scale residual network, such as Figure 2 As shown, it consists of three parts: shallow image feature extraction network, deep image feature extraction network, and up-sampling image reconstruction network. ;
[0042] The shallow feature extraction module is used to extract shallow features of medical fundus image information. It uses a 3×3 convolutional layer to extract features from low-resolution text images, maps RGB channel information into feature space, and transmits this information to the deep image feature extraction module. This feature extraction module only extracts initial features from low-resolution images to obtain shallow features of medical fundus images.
[0043] The deep image feature extraction module is used to extract the deep features of the medical fundus image. The low-resolution medical fundus image features extracted in the previous step are input into the deep feature extraction. The module includes 6 DSMM residual blocks. Each DSMM residual block uses two upper and lower paths and different numbers of DCBK hole convolution blocks and LKA large kernel convolution attention blocks to extract the deep-level fundus medical image texture, and uses residual connections to prevent gradient disappearance.
[0044] The upsampling module inputs the features extracted from the deep image features into the upsampling layer, amplifies the image by 4 times, and obtains a high-resolution medical fundus image. After passing through a 3×3 convolution block, the image is finally reconstructed and the final super-resolution medical fundus image is output;
[0045] It should be noted that the experiments in this paper are conducted on the fundus dataset STARE, which is commonly used in capillary segmentation algorithms. The dataset contains 397 retinal images with a resolution of 700×605 pixels. Since the image resolutions in the dataset are not all multiples of 4, the images need to be preprocessed first. We use MATLAB to resize the images to 696×600 so that they can be used as ground-truth images. Next, we downsample the images to generate images with a resolution of 4 times smaller and ensure that they are consistent with the original images. Figure 1 The experiment selected the first 100 images as the training set and the remaining 297 images as the test set.
[0046] In order to further implement the above technical solution, the residual module includes:
[0047] This module adopts a multi-scale cross network structure, in which one path is dedicated to extracting low-frequency information of the image, and the other path is responsible for extracting high-frequency information. The high-frequency branch of DSMM consists of two dilated convolution blocks DCBK, while the low-frequency information branch consists of a single dilated convolution block. Finally, this fused information is input into the large-core attention block LKA. LKA combines three parts: local spatial convolution, spatial long-range convolution, and channel convolution, and can achieve a 21×21 receptive field. It not only retains the advantages of large-core convolution, but also overcomes the computational complexity and memory capacity requirements of the self-attention mechanism, and extracts the deep information of the image more delicately, thereby showing excellent performance in visual tasks.
[0048] In order to further implement the above technical solution, the loss function includes:
[0049] (1) Mean square error loss
[0050] The mean square error loss, also known as L1 loss, is one of the most common loss functions. It compares the difference between the generated high-resolution image and the real high-resolution image by calculating the average of the squared differences between the pixels:
[0051]
[0052] Where m is the number of samples, y i is the true value of the i-th sample, is the predicted value of the ith sample.
[0053] (2) The overall loss function can be expressed as follows:
[0054] In medical image reconstruction, since single pixel loss makes it difficult to effectively reconstruct high-frequency information, we introduce frequency constraints to regulate network training to achieve better reconstruction results. The total loss function of this algorithm includes a frequency constraint term, and its formula is as follows:
[0055]
[0056] in represents Fourier transform, λ is the weight, usually defaulted to 0.1. Introducing a frequency-constrained loss function can accelerate the convergence of the model and make the spectrum of the generated image closer to the spectrum of the real image.
[0057] It should be noted that:
[0058] Since the network of the present invention is trained end-to-end, the super-resolution result I SR , original high-resolution image I HR , input into the pixel loss function and continuously iterate to minimize the loss function, obtain a set of weight parameters that minimize the total loss function, use this set of parameters as the trained model parameters, and obtain a trained medical fundus image super-resolution model.
[0059] The medical fundus super-resolution reconstruction method based on a multi-level multi-scale residual network includes the following steps:
[0060] S1. Download the fundus dataset STARE for capillary segmentation algorithm. During training, the dataset contains 397 retinal images with a resolution of 700×605 pixels. Since the image resolutions in the dataset are not all multiples of 4, the images need to be preprocessed first. We use MATLAB to resize the images to 696×600 so that they can be used as ground-truth images. Next, we downsample the images to generate images with a resolution of 4 times smaller and ensure that they are consistent with the original images. Figure 1 One-to-one correspondence. The experiment selected the first 100 images as the training set and the remaining 297 images as the test set;
[0061] S2. Input the processed image obtained in S1 into the shallow feature extraction module to extract the shallow features of the medical fundus super-resolution image, and use the convolution layer to extract features of the low-resolution medical fundus image. This convolution layer can only extract the shallow features of the medical fundus image to obtain the low-resolution image I. SR ;
[0062] S3. Input the low-resolution image obtained in S2 into the deep feature extraction module, and input the extracted shallow features of the low-resolution medical fundus image into the DSMM residual module for deep feature extraction. The module includes 6 residual blocks. This module adopts a multi-scale cross network structure, in which one path is specifically used to extract the low-frequency information of the image, and the other path is responsible for the extraction of high-frequency information. The high-frequency branch of DSMM consists of two dilated convolution blocks DCBK, while the low-frequency information branch consists of a single dilated convolution block. Finally, this fused information is input into the large core attention block LKA;
[0063] S4. Input the deep image features obtained in S3 into the upsampling network, input the features extracted from the deep image features into the upsampling layer, magnify the image by 4 times, and obtain a high-resolution medical fundus image. Then, after a 3×3 convolution block, the image is reconstructed and the final super-resolution medical fundus image is output;
[0064] S5. Input the high-resolution image generated in S4 into the 9×9 convolution kernel, perform the final reconstruction on the image, and output the final super-resolution image I SR ;
[0065] S6. Set the hyperparameters of the medical fundus image super-resolution network model, and set the batch size to 16. The optimizer uses the Adam algorithm, where β1=0.9, β2=0.99. The initial learning rate is set to 0.002, the minimum learning rate is 0.0000001, and the cosine annealing strategy is used to update the learning rate. The input image is cropped into small blocks of 32×32 resolution, and then image enhancement operations such as flipping are performed. Finally, the corresponding high-resolution image is output through the network for back propagation. The experiment uses PSNR and SSIM values as objective evaluation criteria.
[0066] The present invention will be further described below through specific experiments:
[0067] 1. Dataset
[0068] The experiments in this paper are conducted on the fundus dataset STARE, which is commonly used in capillary segmentation algorithms. The dataset contains 397 retinal images with a resolution of 700×605 pixels. Since the image resolutions in the dataset are not all multiples of 4, the images need to be preprocessed first. We use MATLAB to resize the images to 696×600 so that they can be used as ground-truth images. Next, we downsample the images to generate images with a resolution of 4 times smaller and ensure that they are consistent with the original images. Figure 1 The experiment selected the first 100 images as the training set and the remaining 297 images as the test set.
[0069] 2. Training details
[0070] The experimental environment configuration of this algorithm is as follows: Intel Core i5-13600KF CPU@3.50GHz, NVIDIARTX4090GPU, and deep learning framework Pytorch is used for experiments. Batch size is set to 16. The optimizer uses the Adam algorithm, where β1=0.9, β2=0.99. The initial learning rate is set to 0.002, the minimum learning rate is 0.0000001, and the cosine annealing strategy is used to update the learning rate. The input image is cropped into small blocks of 32×32 resolution, and then image enhancement operations such as flipping are performed. Finally, the corresponding high-resolution image is output through the network for back propagation. The experiment uses PSNR and SSIM values as objective evaluation criteria, where the higher the PSNR value, the better the image quality, and the closer the SSIM value is to 1, the closer the image is to the original image.
[0071] 3. Results Analysis
[0072] A. Quantitative Analysis
[0073] The proposed algorithm is compared with classic super-resolution reconstruction algorithms such as Bicubic, SRCNN, FSRCNN, VDSR, and IMDN. As can be seen from Table 1, the proposed algorithm achieves excellent performance on the test dataset when the magnification is 4. The experimental results show that compared with SRCNN and FSRCNN, the proposed algorithm has an average improvement of 4.22dB and 2.81dB in PSNR, and the SSIM value is also slightly improved. The peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of image reconstruction quality evaluation indicators are used as auxiliary evaluation indicators. Among them, PSNR represents the difference between the reconstructed image and the original image. The higher the PSNR, the smaller the difference and the higher the image quality; SSIM is a global image quality evaluation indicator that takes into account three aspects: brightness, contrast, and structure. It calculates the similarity between the original image and the reconstructed image. The result is between [-1,1]. The closer the SSIM value is to 1, the more similar the reconstructed image is to the original image.
[0074] Table 1
[0075]
[0076] B. Qualitative analysis
[0077] This article selects two representative pictures for comparison. In the high-frequency information area, the reconstruction effect is as follows Figure 5As shown. It can be seen that in the dense blood vessel area, that is, the high-frequency information part, the edge contours of the IMDN algorithm and the algorithm in this paper are significantly different from those of other algorithms. Except for these two algorithms, other traditional algorithms cannot clearly restore the texture information of capillaries. The clarity of the algorithm in this paper is improved compared with the IMDN algorithm. In the low-frequency information area, the representative image after reconstruction Figure 6 It shows that except for the proposed algorithm and IMDN algorithm, the lines of other algorithms in the retinal edge area have many jagged shapes. The IMDN algorithm has a slightly heavier smearing feeling when reconstructing the dark area of the retina than the proposed algorithm. Whether it is PSNR value or visual perception, the proposed algorithm is better than other lightweight algorithms compared.
[0078] 4. Ablation experiment
[0079] In order to explore how to achieve the best model performance and produce a higher quality fundus super-resolution image network, we conducted several ablation experiments:
[0080] The results of the ablation experiment are shown in Table 2. When the DCBK blocks proposed in this algorithm are reduced from 6 to 4, the number of parameters of the algorithm will be reduced to 467K. Although the final PSNR value decreases, this makes the algorithm more lightweight. If you need to reduce the inference time and the number of parameters, you can set the DCBK block to 4. Changes in the activation function and loss function will lead to performance degradation, so it is correct to choose the GELU activation function and the LKA large-core attention block. The control variables of the loss function show that the L loss function we use can have a certain impact on the results and convergence speed. Finally, we tried to train the model on a natural dataset, and the results were not better than those we trained on the STARE dataset. It can be seen that natural datasets are not universal for training medical image models and may reduce performance.
[0081] Table 2 Ablation experiments on DCBN
[0082]
[0083] In summary, the DCBN algorithm proposed in the present invention is based on dilated separable convolution and multi-scale neural network, which solves the problem of high-frequency and low-frequency details loss in fundus medical image reconstruction in traditional super-resolution methods. Dilated convolution expands the receptive field without increasing parameters to extract information more efficiently, while multi-scale neural network can extract high-frequency and low-frequency information of images more effectively. Separable convolution can alleviate the grid effect caused by dilated convolution. Introducing constrained frequency can not only accelerate model convergence, but also improve model performance. The LKA large-core attention block obtains a larger receptive field with a smaller number of parameters, thereby improving the final performance. After experimental verification, the algorithm proposed in this paper has achieved significant improvement in performance indicators compared with traditional lightweight super-resolution reconstruction algorithms, and has improved the perception of high-frequency and low-frequency details in visual effects. However, the number of parameters of the model in this paper is still slightly larger than that of the above model, reaching 706K, which may affect the inference speed. Although the number of parameters can be reduced by reducing the DCBK residual block, this may also lead to performance degradation. Therefore, future research will focus on how to maintain or improve performance while reducing the number of parameters to reduce the consumption of time and resources.
Claims
1. A method for fusing a multi-level and multi-scale residual structure neural network with a deep separable dilated convolution. The model consists of three parts: a shallow image feature extraction network, a deep image feature extraction network, and an upsampling image reconstruction network. The shallow feature extraction module is used to extract shallow features of medical fundus image information. It uses a 3×3 convolutional layer to extract features from low-resolution text images, maps RGB channel information into feature space, and transmits this information to the deep image feature extraction module. This feature extraction module only extracts initial features from low-resolution images to obtain shallow features of medical fundus images. The deep image feature extraction module is used to extract the deep features of the medical fundus image. The low-resolution medical fundus image features extracted in the previous step are input into the deep feature extraction. The module includes 6 DSMM residual blocks. Each DSMM residual block uses two upper and lower paths and different numbers of DCBK hole convolution blocks and LKA large kernel convolution attention blocks to extract the deep-level fundus medical image texture, and uses residual connections to prevent gradient disappearance. The upsampling module inputs the features extracted from the deep image features into the upsampling layer, amplifies the image by 4 times, and obtains a high-resolution medical fundus image. After passing through a 3×3 convolution block, the image is finally reconstructed and the final super-resolution medical fundus image is output.
2. According to the method based on multi-scale residual structure neural network and deep separable dilated convolution fusion according to claim 1, its residual module is specifically: The extracted shallow features of low-resolution medical fundus images are input into the DSMM residual module for deep feature extraction. The module includes 6 residual blocks. This module adopts a multi-scale cross network structure, in which one path is dedicated to extracting low-frequency information of the image, and the other path is responsible for extracting high-frequency information. The high-frequency branch of DSMM consists of two dilated convolution blocks DCBK, while the low-frequency information branch consists of a single dilated convolution block. Finally, this fused information is input into the large-core attention block LKA.
3. The method based on fusion of multi-scale residual structure neural network and deep separable dilated convolution according to claim 1 is characterized in that: A multi-scale residual network and a dilated convolution block were used. The network adopts a multi-scale cross network structure, in which one path is dedicated to extracting low-frequency information of the image, and the other path is responsible for extracting high-frequency information. Compared with traditional neural networks, it explores new features while also sharing some common features. This structure can more effectively extract high-frequency image features and low-frequency image feature information from fundus medical image reconstruction. Dilated convolution, also known as dilated convolution or expanded convolution) is a special convolution operation that defines the interval between the convolution kernel and each value when processing data by introducing a hyperparameter called "dilation rate". Different from traditional convolution, dilated convolution can expand the receptive field while keeping the size of the output feature map unchanged. The loss functions used in the training of the medical fundus image super-resolution reconstruction network in the steps include: (1) Mean square error loss The mean square error loss, also known as L1 loss, is one of the most common loss functions. It compares the difference between the generated high-resolution image and the real high-resolution image by calculating the average of the squared differences between the pixels: Where m is the number of samples, y i is the true value of the i-th sample, is the predicted value of the ith sample. (2) The overall loss function can be expressed as follows: In medical image reconstruction, since single pixel loss makes it difficult to effectively reconstruct high-frequency information, we introduce frequency constraints to regulate network training to achieve better reconstruction results. The total loss function of this algorithm includes a frequency constraint term, and its formula is as follows: in represents Fourier transform, λ is the weight, usually defaulted to 0.
1. Introducing a frequency-constrained loss function can accelerate the convergence of the model and make the spectrum of the generated image closer to the spectrum of the real image.
4. The method based on fusion of multi-scale residual structure neural network and deep separable dilated convolution according to claim 1 is characterized in that: The following steps are involved: S1. Download the fundus dataset STARE for capillary segmentation algorithm. During training, the dataset contains 397 retinal images with a resolution of 700×605 pixels. Since the image resolutions in the dataset are not all multiples of 4, the images need to be preprocessed first. We use MATLAB to resize the images to 696×600 so that these images can be used as ground-truth images. Next, we downsample the images to generate images with a resolution of 4 times smaller, and ensure that they correspond one-to-one with the original images. The experiment selects the first 100 images as the training set and the remaining 297 as the test set; S2. Input the processed image obtained in S1 into the shallow feature extraction module to extract the shallow features of the medical fundus super-resolution image, and use the convolution layer to extract features of the low-resolution medical fundus image. This convolution layer can only extract the shallow features of the medical fundus image to obtain the low-resolution image I. SR ; S3. Input the low-resolution image obtained in S2 into the deep feature extraction module, and input the extracted shallow features of the low-resolution medical fundus image into the DSMM residual module for deep feature extraction. The module includes 6 residual blocks. This module adopts a multi-scale cross network structure, in which one path is specifically used to extract the low-frequency information of the image, and the other path is responsible for the extraction of high-frequency information. The high-frequency branch of DSMM consists of two dilated convolution blocks DCBK, while the low-frequency information branch consists of a single dilated convolution block. Finally, this fused information is input into the large core attention block LKA; S4. Input the deep image features obtained in S3 into the upsampling network, input the features extracted from the deep image features into the upsampling layer, magnify the image by 4 times, and obtain a high-resolution medical fundus image. Then, after a 3×3 convolution block, the image is reconstructed and the final super-resolution medical fundus image is output; S5. Input the high-resolution image generated in S4 into the 9×9 convolution kernel, perform the final reconstruction on the image, and output the final super-resolution image I SR ; S6. Set the hyperparameters of the medical fundus image super-resolution network model, and set the batch size to 16. The optimizer uses the Adam algorithm, where β1=0.9, β2=0.
99. The initial learning rate is set to 0.002, the minimum learning rate is 0.0000001, and the cosine annealing strategy is used to update the learning rate. The input image is cropped into small blocks of 32×32 resolution, and then image enhancement operations such as flipping are performed. Finally, the corresponding high-resolution image is output through the network for back propagation. The experiment uses PSNR and SSIM values as objective evaluation criteria.
Citation Information
Cited By
Lightweight video super-division method and system based on super-network dynamic pruning
CN120580232A
Lightweight video super-resolution method and system based on hypernetwork dynamic pruning
CN120580232B
