Real world image super-resolution reconstruction method based on super-pixel attention mechanism

Abstract

The invention belongs to the technical field of deep learning, and particularly relates to a real-world image super-resolution reconstruction method based on a super-pixel attention mechanism, and the method comprises the following steps: constructing a data set; an image super-division model SPGAN is constructed; training is carried out on the constructed SPGAN model by using the training set to which the SPGAN model belongs; acquiring a real-world low-resolution image to be subjected to super-resolution reconstruction, and performing real-world image super-resolution reconstruction through the trained SPGAN model; and establishing an evaluation index for evaluating the SPGAN model. According to the invention, a superpixel module SPI is introduced, and the SPI module comprises a superpixel segmentation module, a superpixel cross attention module, a superpixel internal attention module and a local attention module which are integrally connected by using a residual structure, so that the perceptual ability of the model to edge pixels is enhanced, and the problems of fuzzy artifacts and the like of a reconstructed image are reduced; and the super-resolution reconstruction effect of the model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a real-world image super-resolution reconstruction method based on a super-pixel attention mechanism. Background Art

[0002] Real-world image super-resolution (RSISR) is an important research topic in computer vision. In real life, there are a large number of devices that collect real-world images. However, due to limitations of factors such as device performance, the collected real-world images are often low-resolution images (LR) of insufficient quality, with problems such as loss of details, inaccurate colors, and compression artifacts, which seriously affect the results of computer vision tasks such as medical imaging, satellite remote sensing, video surveillance, and traffic detection. Therefore, it is of great practical significance to study how to restore high-resolution images (HR) from low-resolution images (LR) obtained in the real world.

[0003] Real-world image super-resolution (RSISR) is different from ordinary single image super-resolution (SISR). The degradation process of images in the real world is very complex, including blur, noise, distortion, etc., and the degraded low-resolution images may not retain the detailed features such as texture of real-world images. At the same time, it is difficult to collect well-aligned low-resolution and high-resolution image pairs (LR-HR) in real-world image super-resolution tasks, which makes it difficult for some supervised learning methods based on image pairs to obtain good super-resolution results, and the generalization effect is also relatively poor.

[0004] Recently, methods based on degradation modeling have made great progress in some real-world super-resolution tasks. The HR images are degraded using artificially designed degradation models to generate corresponding LR images, and then the HR-LR image pairs are used for model training. The core of this method is to simulate the degradation process of images from high resolution to low resolution in the real world, including the influence of factors such as blur, compression, and noise. In this way, a large number of synthetic datasets can be generated for training and optimizing super-resolution models. These synthetic datasets can not only provide sufficient training samples, but also simulate different degradation scenarios to enhance the generalization ability of the model.

[0005] In practical applications, degradation modeling methods can significantly improve the performance of super-resolution reconstruction. For example, by mixing degradation operations and setting different degradation parameters, LR images with a wide range of degradation effects can be generated. The advantage of this method is that it can easily obtain a large number of paired degraded LR images without the need to collect them laboriously or endure the misalignment problem of unpaired training data. In addition, by changing the degradation parameters, multiple degradation models can be easily obtained, thereby improving the adaptability and robustness of the model in different scenarios.

[0006] With the development of deep learning technology, real-world super-resolution technology based on deep learning has also been actively explored and developed. Various real-world image super-resolution methods based on deep learning have been proposed, and good reconstruction results have been achieved on public datasets. These methods can effectively learn the high-frequency information in the image, including structure, texture, etc., by learning the mapping relationship of the image, thereby restoring rich details and structures in low-resolution images. Therefore, super-resolution technology based on degradation modeling has broad prospects in practical applications, especially in the fields of medical diagnosis, remote sensing images, computer vision research, etc. Summary of the invention

[0007] In response to the above technical problems, the present invention provides a real-world image super-resolution reconstruction method based on a superpixel attention mechanism, which can generate low-resolution degraded images that are closer to the degradation characteristics of the real world, thereby obtaining better super-resolution performance in real-world scenes. At the same time, a real-world image super-resolution method based on superpixels is proposed, which integrates the network architecture of GAN and uses superpixel modules to effectively strengthen the model's attention to image edge pixels, further improving the precision of the super-resolution results.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] The real-world image super-resolution reconstruction method based on superpixel attention mechanism includes the following steps:

[0010] S1. Using GAN as the basic structure, we integrate the superpixel attention mechanism into the generator network to build a real-world image super-resolution model SPGAN.

[0011] S2, introduces a two-stage gated degradation model, and obtains the real-world degraded image by adjusting the blur kernel, size, noise type and performing JPEG compression;

[0012] S3, design a superpixel module in combination with superpixel attention to group similar pixels together perceptually;

[0013] S4. Combine L1 loss, perceptual loss, total variation regularization and GAN loss to train SPGAN, and use exponential moving average EMA to obtain more stable training and better performance.

[0014] The main framework of the real-world image super-resolution model SPGAN in S1 is a GAN structure, in which the generator is used for the super-resolution process of the degraded low-resolution image LR, and the discriminator is responsible for judging the relative authenticity of the super-resolution SR image and the high-resolution HR image.

[0015] The real-world image super-resolution model SPGAN performs super-resolution reconstruction in the following way: a low-resolution image degraded by two-stage gating is passed into a generator, features of the image are captured by N super-pixel blocks SPI, and the reconstruction quality is improved by a PixelShuffle upsampling operation, and the final SR image is obtained by combining the overall residual structure; finally, the SR image and its corresponding HR image are passed into a discriminator to determine the authenticity of the image, and the super-resolution reconstruction effect of the image is improved through adversarial training between the two.

[0016] The method of introducing the two-stage gated degradation model in S2 is: the two-stage gated degradation model includes two identical stages, each of which includes four parts: blur kernel, resizing, adding noise and JPEG compression; the mathematical model can be expressed as:

[0017]

[0018] in represents the second-order gated degradation model, where D 1 and D 2 Denote the degradation operations of the first and second stages respectively, D k represents blur kernel degradation, D s represents downsampling degradation, D n represents noise degradation, D j represents JPEG compression degradation; σ represents the gating probability, which is used to control whether the corresponding degradation operation takes effect; the gating operation mechanism is integrated into the above four parts to achieve flexible regulation of the intensity of the degradation operation, so as to more accurately simulate the actual degradation phenomenon.

[0019] The method of designing a superpixel module in combination with superpixel attention in S3 is as follows: including superpixel segmentation, superpixel cross attention, superpixel internal attention, and a local attention module, and the overall residual structure is used to simplify training; the superpixel internal attention uses two cascaded linear layers to replace the two matrix multiplication operations in the self-attention operation, so that the model has lower complexity and better performance; multiple superpixel attention modules in parallel can better enhance the model's attention to local and global features.

[0020] The method for training SPGAN in S4 by combining L1 loss, perceptual loss, total variation regularization and GAN loss is:

[0021] First, L1 loss is used to calculate the average error between the super-resolution image and the real image. The loss formula is defined as follows:

[0022]

[0023] Where n is the total number of samples, f(x) is the super-resolution image, and yi For real-world images, the L1 loss function is calculated by calculating f(x) and y i The absolute value of the difference is used to iteratively optimize the model;

[0024] Train a PSNR-oriented model, and the trained model is named SPNet;

[0025] We then use the trained PSNR-oriented model as the generator initialization and train SPGAN with a combination of L1 loss, perceptual loss, total variation regularization, and GAN loss.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention proposes a real-world image super-resolution reconstruction method SPGAN based on super-pixel attention mechanism. A two-stage gated degradation model is proposed, which makes the low-resolution image after model degradation closer to the real-world degraded image, and reduces the unsatisfactory super-resolution effect caused by the large gap between the model-degraded image and the real-world degraded image to a certain extent, further improving the real-world image super-resolution effect. At the same time, combined with the GAN network structure, the super-pixel module is applied to the generator network, so that the model can pay more attention to the edge pixels of the image, which solves the problems of image blurring artifacts caused by the GAN network to a certain extent, and improves the fineness of the super-resolution image. The pre-trained model SPNet network in the present invention is quantitatively compared on the Set14 data set. When the noise level is 15, a PSNR value of 23.74 and a SSIM value of 0.6165 are obtained. When the noise level is 25, a PSNR value of 23.03 and a SSIM value of 0.578 are obtained. A quantitative comparison is performed on the BSD100 data set. When the noise level is 15, a PSNR value of 23.83 and a SSIM value of 0.6176 are obtained. When the noise level is 25, a PSNR value of 23.31 and a SSIM value of 0.5864 are obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0029] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0030] Figure 1 The overall architecture diagram of the SPGAN of the present invention is shown;

[0031] Figure 2 It is a structural diagram of super pixel blocks in the SPGAN model of the present invention;

[0032] Figure 3 This is a comparison chart of the experimental results of the SPNet of the present invention and excellent real-world super-resolution methods on synthetic images;

[0033] Figure 4 This is a comparison chart of the experimental results of the SPGAN of the present invention and excellent real-world super-resolution methods on real-world images. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0035] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0036] The present invention is implemented under the pytorch deep learning framework. The present invention provides a real-world image super-resolution reconstruction method based on a superpixel attention mechanism, which specifically includes the following steps:

[0037] 1. Data preparation

[0038] The data sample of the present invention is the DF2K image super-resolution dataset:

[0039] The DF2K dataset is a combination of the DIV2K dataset and the Flickr2K dataset. The DIV2K dataset contains 1,000 low-resolution images of various degradation types, including 800 training images, 100 verification images, and 100 test images; the Flickr2K dataset contains 2,650 2K resolution images, including people, animals, landscapes, and other types, covering a variety of natural scenes and daily objects.

[0040] During training, images are randomly cropped to a size of 256×256.

[0041] 2. Model construction

[0042] The main framework of the constructed SPGAN model is a GAN structure. The specific network structure is as follows: Figure 1 As shown in Figure 1, the SPGAN model consists of two modules: the generator and the discriminator. The generator first uses a 3×3 convolutional layer to perform low-resolution image I LR Then, the encoder is constructed using n super-pixel modules. Figure 2 As shown in Figure 1, each superpixel module consists of four parts: superpixel segmentation module, superpixel cross attention module, superpixel internal attention module, and local attention module. In terms of the details of superpixel block construction, we first pass the pixel information to the superpixel layer through the linear layer to generate the query vector Q required by the self-attention mechanism. s , key vector K I Sum value vector V I These vectors are connected with the corresponding weight matrix and Next, we calculate the query vector Q s With the key vector K I and use this similarity as the attention weight, and the value vector V I Multiply them together to get the updated superpixel features. This process can be expressed as:

[0043]

[0044] in, is a scaling factor used to prevent gradient vanishing. The updated superpixel features are then mapped back to the pixel level through a cross-attention mechanism. This step solves the problem of long-range information transmission between superpixels and ensures the effective flow of information between pixels. After passing through n superpixel blocks, the extracted features are then passed through a 3×3 convolution layer and an upsampling Pixel Shuffle layer, and the result is finally compared with the low-resolution image I LRResidual fusion is performed; the discriminator adopts the U-Net structure design with stronger identification ability, which can provide more detailed identification feedback. At the same time, in order to increase the training stability, the spectral normalization regularization method is used to stabilize the training process.

[0045] 3. Model training

[0046] The training process adopts two stages.

[0047] First, we use L1 loss to calculate the average error between the super-resolution image and the real image. The loss formula is defined as follows:

[0048]

[0049] Where n is the total number of samples, f(x) is the super-resolution image, and y i For real-world images, the L1 loss function is calculated by calculating f(x) and y i The absolute value of the difference is used to iteratively optimize the model.

[0050] A PSNR-oriented model is trained and named SPNet.

[0051] We then use the trained PSNR-oriented model as the generator initialization and train SPGAN with a combination of L1 loss, perceptual loss, total variation regularization, and GAN loss.

[0052] 4. Test results

[0053] The method for training super-resolution reconstruction from low-resolution images is as follows: for real-world image datasets, the resolution of the test images in the dataset is adjusted to 256×256 using bicubic interpolation as HR images; the designed two-stage gated degradation model is used to degrade the HR images to 16×16 resolution images as LR images. First, the trained SPNet model is used to reconstruct the LR images to obtain SR images, and the SSIM and PNSR are used as performance evaluation indicators to evaluate the reconstruction effect; then the SPGAN model is used to super-reconstruct the LR images to obtain SR images, and NIQE is used as the performance evaluation indicator to evaluate the reconstruction effect.

[0054] 5. Model Evaluation

[0055] The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) evaluation indicators are calculated using super-resolution results and real images to evaluate the performance of the SPNet model.

[0056] Table 1 Comparison results of different models on Set14 and BSD100 datasets

[0057]

[0058] The experiment first compared SPNet with four advanced real-world image super-resolution algorithms, namely BSRNet, DASR, Real-SwinIR-L and Real-ESRNet. The quantitative comparative super-resolution image effect evaluation was carried out on the Set14 dataset and the BSD100 dataset. The optimal indicators in the table are in bold.

[0059] Table 1 shows that under the same experimental environment, SPNet performs better than the comparison algorithms in PSMR and SSIM results in Set14 and BSD100 datasets. In Set14 dataset, under two noise levels, PSNR is significantly improved compared to BSRNet and DASR methods, and slightly improved compared to Real-SwinIR-L, and the performance is almost the same as that of Real-ESRNet model; SSIM is also very close to Real-ESRNet. In BSD100 dataset, it performs well under two noise levels. PSNR is still significantly improved compared to BSRNet and DASR methods, and slightly improved compared to Real-SwinIR-L; compared to Real-ESRNet, PSNR and SSIM are very close. Figure 3 The comparison results show that SPNet has better reconstruction effect on severely degraded images than the current excellent real-world super-resolution model.

[0060] The super-resolution results and real images are used to calculate the NIQE (Natural Image Quality Evaluator) score to evaluate the performance of the SPGAN model.

[0061] Table 2 Comparison results of different models on various real-world image datasets

[0062]

[0063] Table 2 compares the SPGAN model with four advanced real-world image super-resolution methods, namely Bicubic, ESRGAN, CDC, and BSRGAN. The quantitative comparison of super-resolution image effects is evaluated on the RealSR-Canon dataset, OST300 dataset, and ADE20Kval dataset. The optimal indicators in the table are in bold.

[0064] The comparison results in Table 2 show that SPGAN outperforms most real-world super-resolution methods on the three datasets. On the RealSR-Canon dataset and OST300 dataset, the NIQE scores of the SPGAN model are 5.6214 and 3.2413 respectively, and the reconstruction effect is significantly better than other real-world super-resolution methods. On the ADE20Kval dataset, the NIQE score of the SPGAN model is also slightly better than other excellent super-resolution models. This shows that SPGAN can better cope with various degradation situations in the real world and reconstruct LR images into high-quality SR images to the greatest extent. Figure 4 From the comparison, we can find that SPGAN not only has a significant improvement in improving image resolution, but also can well preserve detail information such as edge contours, textures, etc., which shows that SPGAN has superior performance in real-world image super-resolution tasks.

[0065] Only the preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the protection scope of the present invention.

Claims

1. A real-world image super-resolution reconstruction method based on superpixel attention mechanism, characterized in that: The following steps are involved: S1. Using GAN as the basic structure, we integrate the superpixel attention mechanism into the generator network to build a real-world image super-resolution model SPGAN. S2, introduces a two-stage gated degradation model, and obtains the real-world degraded image by adjusting the blur kernel, size, noise type and performing JPEG compression; S3, design a superpixel module in combination with superpixel attention to group similar pixels together perceptually; S4. Combine L1 loss, perceptual loss, total variation regularization and GAN loss to train SPGAN, and use exponential moving average EMA to obtain more stable training and better performance.

2. The real-world image super-resolution reconstruction method based on superpixel attention mechanism according to claim 1, characterized in that: The main framework of the real-world image super-resolution model SPGAN in S1 is a GAN structure, in which the generator is used for the super-resolution process of the degraded low-resolution image LR, and the discriminator is responsible for judging the relative authenticity of the super-resolution SR image and the high-resolution HR image.

3. The real-world image super-resolution reconstruction method based on superpixel attention mechanism according to claim 2, characterized in that: The real-world image super-resolution model SPGAN performs super-resolution reconstruction in the following way: a low-resolution image degraded by two-stage gating is passed into a generator, features of the image are captured by N super-pixel blocks SPI, and the reconstruction quality is improved by a PixelShuffle upsampling operation, and the final SR image is obtained by combining the overall residual structure; finally, the SR image and its corresponding HR image are passed into a discriminator to determine the authenticity of the image, and the super-resolution reconstruction effect of the image is improved through adversarial training between the two.

4. The real-world image super-resolution reconstruction method based on superpixel attention mechanism according to claim 1, characterized in that: The method of introducing the two-stage gated degradation model in S2 is: the two-stage gated degradation model includes two identical stages, each of which includes four parts: blur kernel, resizing, adding noise and JPEG compression; the mathematical model can be expressed as: in represents the second-order gated degradation model, where D 1 and D 2 Denote the degradation operations of the first and second stages respectively, D k represents blur kernel degradation, D s represents downsampling degradation, D n represents noise degradation, D j represents JPEG compression degradation; σ represents the gating probability, which is used to control whether the corresponding degradation operation takes effect; the gating operation mechanism is integrated into the above four parts to achieve flexible regulation of the intensity of the degradation operation, so as to more accurately simulate the actual degradation phenomenon.

5. The real-world image super-resolution reconstruction method based on superpixel attention mechanism according to claim 1, characterized in that: The method of designing a superpixel module in combination with superpixel attention in S3 is as follows: including superpixel segmentation, superpixel cross attention, superpixel internal attention, and a local attention module, and the overall residual structure is used to simplify training; the superpixel internal attention uses two cascaded linear layers to replace the two matrix multiplication operations in the self-attention operation, so that the model has lower complexity and better performance; multiple superpixel attention modules in parallel can better enhance the model's attention to local and global features.

6. The real-world image super-resolution reconstruction method based on superpixel attention mechanism according to claim 1, characterized in that: The method for training SPGAN in S4 by combining L1 loss, perceptual loss, total variation regularization and GAN loss is: First, L1 loss is used to calculate the average error between the super-resolution image and the real image. The loss formula is defined as follows: Where n is the total number of samples, f(x) is the super-resolution image, and y i For real-world images, the L1 loss function is calculated by f(x) and y i The absolute value of the difference is used to iteratively optimize the model; Train a PSNR-oriented model, and the trained model is named SPNet; We then use the trained PSNR-oriented model as the generator initialization and train SPGAN with a combination of L1 loss, perceptual loss, total variation regularization, and GAN loss.

Images