Blind super-resolution reconstruction algorithm for dynamic fuzzy kernel estimation

Abstract

The invention provides a blind super-resolution reconstruction algorithm based on dynamic fuzzy kernel estimation. A dynamic depth linear filtering module is used for estimating a blurring kernel of a low-resolution image for defuzzification, and the module can adaptively generate the weight of the blurring kernel according to the input of the low-resolution image; inputting the estimated blurring kernel and the low-resolution image into a super-division network, and performing shallow feature extraction, deep feature extraction and image reconstruction in sequence; and finally, a reconstructed high-resolution image is obtained through up-sampling. By using the method provided by the invention, a low-resolution medical image can be better reconstructed to obtain a high-resolution medical image.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and medical image processing, and in particular to a blind super-resolution reconstruction algorithm for dynamic blur kernel estimation. Background Art

[0002] With the improvement of modern medical technology, medical imaging technology has been widely used in medical diagnosis, and medical images have become an important auxiliary tool for medical diagnosis. The high-resolution medical images obtained after super-resolution reconstruction are both clear and rich in detail features, providing doctors with detailed and accurate information, which greatly helps doctors to accurately judge the patient's condition.

[0003] Medical image super-resolution reconstruction refers to the process of processing a known low-resolution image based on its information and generating a high-resolution image. In traditional methods, reconstruction methods based on interpolation include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. This method is simple and fast, but the processing effect is poor at pixel mutations such as edges and textures of the reconstructed image. The reconstruction-based method can use prior knowledge to constrain the reconstruction process, but the computational complexity is high and it cannot simulate the real scene well. Compared with natural images, medical images have smaller texture details and are accompanied by a large amount of interference information, which will affect the doctor's judgment. For example, the ghosting caused by the vibration of the patient's internal organs requires that the super-resolution reconstructed image be clearer. Most of the existing medical image processing uses a single scale to extract features, which is not effective in medical image reconstruction.

[0004] In recent years, with the development of deep learning, deep learning methods have been gradually used to handle super-resolution tasks. Deep learning-based methods can be divided into super-resolution methods based on convolutional neural networks (CNN), super-resolution methods based on residual networks (ResNet), super-resolution methods based on generative adversarial networks (GAN), and super-resolution methods based on Transformer (a self-attention mechanism). Transformer-based super-resolution methods have gradually attracted widespread attention from scholars. Chen et al. proposed a Hybrid Attention Transformer model that combines channel attention and self-attention to utilize more input information for reconstruction, and also introduced an overlapping cross-attention module to enhance the interaction of cross-window features. However, the amount of calculation and parameters is still large. Compared with traditional convolution-based methods, it requires more hardware resources and training time. The Deformable Attention Transformer model proposed by Xia et al. uses an adaptive convolution kernel generator to dynamically adjust the size and shape of the convolution kernel, thereby achieving more flexible and accurate feature extraction. However, it has not been fully tested and verified in a variety of data sets and scenarios, so its generalization ability and robustness need to be improved. Zhou et al. proposed a Transformer-based image super-resolution method SRFormer, which uses multi-scale feature fusion and multi-head self-attention mechanism to extract and transfer image features. However, it requires a pre-trained model to train the Transformer network, which may not be suitable for some specific fields or tasks. Liang et al. proposed an image super-resolution method based on Swin Transformer, which uses a local window self-attention mechanism and a sliding window mechanism to effectively capture long-distance dependencies. However, the pre-training strategy requires the use of a larger data set, which may increase the difficulty and cost of data acquisition and processing. Although the above-mentioned deep learning-based image super-resolution algorithms have achieved good reconstruction results on natural images, there are still some problems in the reconstruction of medical images. For example, most of these algorithms add a large number of parameters and computational complexity to achieve better reconstruction effects, but the computational cost is too high in real life, which limits their application. Moreover, the attention mechanism modules of most algorithms are too complex, and the extracted features will have many useless features, which will interfere with the diagnosis made by doctors based on the reconstructed images. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a blind super-resolution reconstruction algorithm based on dynamic fuzzy kernel estimation, which highlights the texture details of medical image reconstruction, improves the performance of feature extraction network, and realizes the reconstruction of low-resolution medical images. The method includes the following steps:

[0006] The technical solution adopted by the present invention is: a blind super-resolution reconstruction algorithm for dynamic blur kernel estimation. First, we first reformulate the degradation model so as to transfer the deblur kernel estimation to the low-resolution space, and introduce a dynamic deep linear filtering module to estimate the blur kernel of the low-resolution image for deblurring. This module can adaptively generate the weights of the blur kernel according to the input of the low-resolution image and produce a more robust kernel estimate; secondly, the estimated blur kernel and the low-resolution image are input into the super-resolution network, and shallow features are extracted through convolution, and deep features are extracted through repeated ResBlock. Finally, a high-quality image is reconstructed by aggregating shallow and deep features and upsampling to obtain a reconstructed high-resolution image. The method comprises the following steps:

[0007] Step 1: 1,000 high-resolution (HR) images were collected from CT, MRI, and X-Ray datasets. The low-resolution images were downsampled to generate corresponding low-resolution (LR) images using a specific degradation kernel (isotropic anisotropic Gaussian kernel) setting and the classic single image super-resolution (SISR) degradation model;

[0008] Step 2: Adaptively generate blur kernel weights based on the input low-resolution image;

[0009] Step 3: Input the low-resolution image into the super-resolution network at the same time, and first perform shallow feature extraction through a layer of convolution;

[0010] Step 4: Use the ResBlock module to extract deep features from shallow features;

[0011] Step 5: Repeat the ResBlock module to extract deep features, and finally pass it through a layer of convolution;

[0012] Step 6: Aggregate the shallow and deep features, and then upsample the aggregated features using a sub-pixel convolution layer to obtain a reconstructed high-resolution image.

[0013] Furthermore, the classic single image super-resolution (SISR) degradation model in step 1 is as follows:

[0014]

[0015] where the input LR image Y is the blurred and downscaled HR image X with additional Gaussian white noise n, K is the blur kernel applied to X, represents a convolution operation, and ↓s represents downsampling by a scaling factor of s.

[0016] Furthermore, in step 2, a dynamic deep linear filter module is introduced to estimate the blur kernel of the low-resolution image. During the training process, the width of the blur kernel is uniformly sampled from a specific range.

[0017] In the dynamic deep linear filter module, we set the filters to 11×11, 7×7, 5×5, and 1×1, making the receptive field 21×21.

[0018] The filters of each layer are generated according to the LR image, and all filters are explicitly convolved sequentially into a single narrow kernel with a stride of 1. Mathematically expressed:

[0019]

[0020] Where Q r Representing the r-th layer filter, we can get the following single-layer kernel function, which is initialized as a 1*1 tensor with a value of 1, and then padded to a size of 21*21. After 11×11, 7×7, 5×5 and 1×1 convolutions, it becomes the estimated kernel. The four convolution weights can be considered as the blur kernel information contained in LR, and then reshaped into an explicit 21*21 blur kernel.

[0021] Furthermore, in step 3, the convolutional layer is used to extract shallow features of the low-resolution image.

[0022] We use a 3×3 convolutional layer H SF (·) Extract shallow features F0:

[0023] F0=H SF (I LQ ) (3)

[0024] Furthermore, the specific processing process of using the ResBlock module in step 4 is as follows:

[0025] Step 4.1: Provides a simple method to map the input image space to a higher dimensional feature space;

[0026] F DF =H DF (F0) (4)

[0027] In formula (4), H DF (·) is a deep feature extraction module, which contains M residual blocks and a convolutional layer. More specifically, the intermediate features F1, F2, F3, …, Fk And output deep features F DF Extract block by block as:

[0028] F i =H RSTBi (F i -1), i=1,2,3…,K F DF =H CONV (F K ) (5)

[0029] In formula (5), H RSTBi (·) represents the i-th residual block, H CONV is the last convolutional layer.

[0030] Step 4.2: The residual block consists of a layer of convolution plus ReLU plus another layer of convolution, and residual connections are used in the whole process to improve the rate of the network.

[0031] Step 5: Use the ResBlock module multiple times to extract deep features, and finally pass it through a layer of convolution.

[0032] Step 6: Reconstruct high-quality image I by aggregating shallow and deep features RHQ :

[0033] I RHQ =H REC (F0+F DF ) (6)

[0034] Among them, H REC (·) represents the estimated blur kernel, which is used to reverse the blur effect in the image degradation process, F DF It is the extracted deep features. The shallow features mainly contain low frequencies, while the deep features focus on restoring the lost high frequencies.

[0035] Furthermore, in order to implement the reconstruction module, we use the sub-pixel convolution layer to upsample the features to obtain a reconstructed high-resolution image. The specific operations are as follows:

[0036] F up =PixelShuffle(C(F)) (7)

[0037] Among them, F up is the upsampled feature map, C represents the convolution operation, and PixelShuffle represents the pixel shading operation. The upsampling method can effectively increase the resolution of the feature map while maintaining the dimension of the feature.

[0038] Furthermore, in step 8, we use Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) to evaluate our experimental results.

[0039] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: the reconstructed degradation model modifies and expands the traditional degradation model to better adapt to the blind super-resolution problem. First, the blur kernel estimation and image upsampling are decoupled, so that the blur kernel can be estimated more accurately in the low-resolution space, thereby avoiding the ambiguity problem caused by the downsampling step. Subsequently, the dynamic deep linear kernel (DDLK) is introduced to improve the kernel estimation, and the blur kernel weight is adaptively generated according to the input low-resolution (LR) image. This means that each image has its own specific blur kernel, thereby improving the accuracy and flexibility of the kernel estimation. At the same time, the obtained blur kernel and the low-resolution image are input into the super-resolution network for shallow and deep feature extraction, and the ResBlock module is reused many times to better improve the rate of the network. Finally, the sub-pixel convolution layer is used to upsample the features to obtain a reconstructed high-resolution image. The present invention can effectively improve the resolution of medical image reconstruction, establish an accurate mapping from low-resolution images to high-resolution images, and thus achieve better reconstruction of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Represents the overall network architecture diagram;

[0041] Figure 2 A diagram showing the structure of DDLK (Dynamic Deep Linear Kernel);

[0042] Figure 3 A structural diagram representing a super-resolution network;

[0043] Figure 4 Represents the structural diagram of the residual block; DETAILED DESCRIPTION

[0044] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation, and the embodiments described below are only a part of the present invention and should not be construed as being limited to all examples of the above-mentioned manner.

[0045] The present invention is mainly aimed at the application requirements of high-resolution reconstruction of medical images. In view of the problems of image blur and insufficient texture information in the existing super-resolution reconstruction of medical images, a super-resolution reconstruction algorithm with dynamic deep linear residual is proposed. First, a dynamic deep linear filtering module is introduced to estimate the blur kernel of the low-resolution image for deblurring. This module can adaptively generate the weight of the blur kernel according to the input of the low-resolution image; secondly, the estimated blur kernel and the low-resolution image are input into the super-resolution network, and shallow feature extraction, deep feature extraction and image reconstruction are performed in turn; finally, the reconstructed high-resolution image is obtained by upsampling.

[0046] Figure 1 It shows the overall framework of the network. Figure 2 , 3 ,4 shows the sub-network structure diagram in the network. The training dataset includes 1,000 high-resolution (HR) images collected from CT, MRI, and X-Ray datasets.

[0047] Step 1: Based on the reconstructed degradation model, a data training set is constructed, including high-resolution images and low-resolution images. The low-resolution images are downsampled to generate corresponding low-resolution (LR) images using a specific degradation kernel (isotropic anisotropic Gaussian kernel) setting and a classic single image super-resolution (SISR) degradation model.

[0048] The classic single image super-resolution (SISR) degradation model is as follows:

[0049]

[0050] where the input LR image Y is the blurred and downscaled HR image X with additional Gaussian white noise n, K is the blur kernel applied to X, represents a convolution operation, and ↓s represents downsampling by a scaling factor of s.

[0051] Step 2: Estimation of blur kernel of low-resolution image: A dynamic deep linear filter module is introduced to estimate blur kernel of low-resolution image. During the training process, the width of blur kernel is uniformly sampled from a specific range. Figure 2 Specifically, the kernel size is fixed to 21 × 21. We uniformly sample the kernel widths for SR scale factors 2, 3, and 4 from the range [0.2, 2.0], [0.2, 3.0], and [0.2, 4.0].

[0052] In the dynamic deep linear filter module, we set the filters to 11×11, 7×7, 5×5, and 1×1, making the receptive field 21×21.

[0053] The filters of each layer are generated according to the LR image, and all filters are explicitly convolved sequentially into a single narrow kernel with a stride of 1. The number of kernels in a single layer is expressed as:

[0054]

[0055] Where Q r Representing the r-th layer filter, we can get the following single-layer kernel function, which is initialized as a 1*1 tensor with a value of 1, and then padded to a size of 21*21. After 11×11, 7×7, 5×5 and 1×1 convolutions, it becomes the estimated kernel. The four convolution weights can be considered as the blur kernel information contained in LR, and then reshaped into an explicit 21*21 blur kernel.

[0056] Step 3: Use the convolutional layer to extract shallow features from the low-resolution image. Figure 3 As shown in Figure 2, shallow feature extraction is performed preferentially in the super-resolution network.

[0057] We use a 3×3 convolutional layer H SF (·) Extract shallow features F0:

[0058] F0=H SF (I LQ ) (3)

[0059] Step 4: Then, the ResBlock module is repeated multiple times in the super-resolution network to perform deep feature extraction.

[0060] Step 4.1: Provides a simple method to map the input image space to a higher dimensional feature space;

[0061] F DF =H DF (F0) (4)

[0062] In formula (4), H DF (·) is a deep feature extraction module, which contains M residual blocks and a convolutional layer. More specifically, the intermediate features F1, F2, F3, …, F k And output deep features F DF Extract block by block as:

[0063] F i =H RSTBi (F i -1), i=1,2,3…,K F DF =H CONV (F K ) (5)

[0064] In formula (5), H RSTBi(·) represents the i-th residual block, H CONV is the last convolutional layer.

[0065] Step 4.2: If Figure 4 As shown in the figure, the residual block consists of a layer of convolution plus ReLU plus another layer of convolution, and residual connections are used in the whole process to improve the rate of the network.

[0066] Step 5: Finally, upsample the aggregated shallow and deep features to reconstruct the high-resolution image I RHQ :

[0067] I RHQ =H REC (F0+F DF ) (6)

[0068] Among them, H REC (·) represents the estimated blur kernel, which is used to reverse the blur effect in the image degradation process, F DF It is the extracted deep features. The shallow features mainly contain low frequencies, while the deep features focus on restoring the lost high frequencies.

[0069] To implement the reconstruction module, we use a sub-pixel convolutional layer to upsample the features to obtain a reconstructed high-resolution image. The specific operations are as follows:

[0070] F up =PixelShuffle(C(F)) (7)

[0071] Among them, F up is the upsampled feature map, C represents the convolution operation, and PixelShuffle represents the pixel shading operation. The upsampling method can effectively increase the resolution of the feature map while maintaining the dimension of the feature.

[0072] Step 6: Input the low-resolution medical image to be reconstructed into the network to obtain a reconstructed high-resolution image.

[0073] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the implementation. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention, and these equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A blind super-resolution reconstruction algorithm for dynamic blur kernel estimation, characterized in that: The following steps are involved: S1: Data preprocessing: 1,000 images are selected from the medical image datasets CT, MRI, and X-Ray for model training. S2: Adaptively generate blur kernel weights based on the input low-resolution image. S3: Shallow feature extraction: The low-resolution image is input into the shallow feature extraction module, and the shallow features of the low-resolution image are obtained through this module. S4: Deep feature extraction network: The low-resolution image is input into the deep feature extraction module, and the deep features of the low-resolution image are obtained through multiple residual blocks and convolution operations in this module. S5: Image reconstruction module: The shallow features and deep features obtained in the previous steps are used as the input of the image reconstruction module, and finally high-resolution image reconstruction is achieved.

2. According to the blind super-resolution reconstruction algorithm for dynamic blur kernel estimation according to claim 1, the step S2 is characterized in that: A dynamic deep linear filter module is introduced to estimate the blur kernel for low-resolution images. During training, the width of the blur kernel is uniformly sampled from a specific range. Specifically, the kernel size is fixed to 21×21. We uniformly sample the kernel widths for SR scale factors 2, 3, and 4 from the ranges [0.2, 2.0], [0.2, 3.0], and [0.2, 4.0]. In the dynamic deep linear filter module, we set the filters to 11×11, 7×7, 5×5, and 1×1, resulting in a receptive field of 21×21. Generate filters for each layer based on the LR image, and explicitly convolve all filters sequentially onto a single narrow kernel with a stride of 1. We can get the following single-layer kernel function, which is initialized as a 1*1 tensor with a value of 1, and then padded to a size of 21*21. After 11×11, 7×7, 5×5 and 1×1 convolutions, it becomes the estimated kernel. The four convolution weights can be considered as the blur kernel information contained in LR, and then reshaped into an explicit 21*21 blur kernel.

3. According to the blind super-resolution reconstruction algorithm for dynamic blur kernel estimation according to claim 1, the step S3 is characterized in that: The preprocessed low-resolution image is input into the shallow feature extraction block, using a 3×3 convolutional layer H SF (·) Extract shallow features F0, the extraction formula is F0 = H SF (I LQ ), and finally we get the shallow features we want for subsequent image reconstruction operations.

4. According to the blind super-resolution reconstruction algorithm for dynamic blur kernel estimation according to claim 1, the step S4 is characterized in that: Inputting low-resolution images into a deep feature extraction network, the convolutional layer excels at early visual processing, leading to more stable optimization and better results. It also provides a simple way to map the input image space to a higher-dimensional feature space, F DF =H DF (F0) where H DF (·) is the deep feature extraction module, which contains M residual blocks and a convolutional layer. More specifically, the intermediate features F1, F2, F3, …, F k And output deep features F DF Extract block by block as F i =H RSTBi (F i -1), i=1,2,3…,K,F DF =H CONV (F K ), where H RSTBi (·) represents the i-th residual block, H CONV It is the last convolutional layer, which finally obtains the deep features of the image.

5. According to the blind super-resolution reconstruction algorithm for dynamic blur kernel estimation according to claim 4, the step S4 is characterized in that: The residual block consists of a layer of convolution plus ReLu plus another layer of convolution, and residual connections are used throughout the process to increase the network speed.

6. According to the blind super-resolution reconstruction algorithm for dynamic blur kernel estimation according to claim 1, the step S5 is characterized in that: We reconstruct high-quality images by aggregating shallow and deep features. RHQ , I RHQ =H REC (F0+F DF ), shallow features mainly contain low frequencies, while deep features focus on recovering the lost high frequencies. In order to implement the reconstruction module, we use a sub-pixel convolutional layer to upsample the features and finally obtain a reconstructed high-resolution image.

Images