Super-resolution image reconstruction method based on improved Real-ESRGAN model

By introducing self-attention module and spectral normalized U-Net structure in the Real-ESRGAN model, the problems of missing image details and low quality in the prior art are solved, and super-resolved with higher quality and robustness are achieved Rate image reconstruction.

CN118918006APending Publication Date: 2024-11-08KUNMING LOGAN KSEC AIRPORT LOGISTICS SYST COMPANY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410912835.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing super-resolution technology based on deep learning has the problem of lack of image details and low reconstruction image quality.

Method used

The improved Real-ESRGAN model is adopted to add the self-attention module and replace RRDB with Z_RRDB. Combined with the U-Net structure with spectral normalization, the total loss function L=Lpercep+λLG+ηL1 is used to perceive the loss, GAN network loss and average absolute error loss are trained to generate networks.

Benefits of technology

Improves the quality and details of reconstructed images, enhances the robustness of the model, better captures the global dependencies of the images and restores details, and generates more realistic and clear high-resolution images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918006A_ABST
    Figure CN118918006A_ABST
Patent Text Reader

Abstract

The invention discloses a super-resolution image reconstruction method based on an improved Real-ESRGAN model, and the method comprises the following steps: inputting a to-be-reconstructed low-resolution image into a pre-trained generative network of the improved Real-ESRGAN model, and obtaining a reconstructed high-resolution image; wherein the generative network of the improved Real-ESRGAN model is additionally provided with a self-attention module. The method can improve the quality of the reconstructed image, enhance the details of the reconstructed image and improve the robustness of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image reconstruction, and more particularly to a super-resolution image reconstruction method based on an improved Real-ESRGAN model. Background Art

[0002] Super-resolution technology aims to reconstruct low-resolution (LR) images into high-resolution (HR) images. The methods for achieving super-resolution can generally be divided into two types: image interpolation-based methods and deep learning-based methods. Image interpolation-based super-resolution technology increases the size of the image by inserting new pixels around the original pixels of the image and assigning values ​​to these new pixels, thereby restoring the image content and improving the image resolution. Super-resolution technology based on deep learning is trained through deep neural networks to learn the mapping relationship from low-resolution images to high-resolution images. This method can extract more information from the image and generate more realistic and clear high-resolution images. With the continuous development of deep learning technology, super-resolution technology based on deep learning has become a mainstream method.

[0003] However, existing super-resolution technologies based on deep learning still suffer from the problem of lack of image details and low quality of reconstructed images.

[0004] Therefore, how to provide a super-resolution image reconstruction method that can effectively enhance image details and improve the quality of reconstructed images is an urgent problem that needs to be solved by those skilled in the art. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a super-resolution image reconstruction method based on an improved Real-ESRGAN model.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A super-resolution image reconstruction method based on an improved Real-ESRGAN model comprises the following steps:

[0008] Input the low-resolution image to be reconstructed into the generative network of the pre-trained improved Real-ESRGAN model to obtain the reconstructed high-resolution image;

[0009] Among them, the generation network of the improved Real-ESRGAN model adds a self-attention module.

[0010] Preferably, the generative network of the improved Real-ESRGAN model replaces all RRDB in the original Real-ESRGAN model generative network with Z_RRDB;

[0011] The Z_RRDB includes a first Conv_LReLU layer, a second Conv_LReLU layer, a third Conv_LReLU layer, a fourth Conv_LReLU layer, a fifth convolutional layer, a self-attention module, a sixth convolutional layer, an element-by-element addition unit, and a seventh convolutional layer connected in sequence;

[0012] The input end of the first Conv_LReLU layer is connected to the input ends of the second Conv_LReLU layer, the third Conv_LReLU layer, the fourth Conv_LReLU layer, and the fifth convolutional layer respectively;

[0013] The input end of the second Conv_LReLU layer is connected to the input ends of the third Conv_LReLU layer, the fourth Conv_LReLU layer, and the fifth convolutional layer respectively;

[0014] The input end of the third Conv_LReLU layer is connected to the input ends of the fourth Conv_LReLU layer and the fifth convolutional layer respectively;

[0015] The output of the fifth convolutional layer is connected to the input of the sixth convolutional layer;

[0016] The input of the first Conv_LReLU layer is connected to the input of the element-wise addition unit.

[0017] Preferably, the discriminant network of the improved Real-ESRGAN model adopts a U-Net structure with spectral normalization.

[0018] Preferably, the total loss function of the generative network of the improved Real-ESRGAN model is:

[0019] L=L percep +λL G +ηL1;

[0020] Wherein, L represents the total loss of the generative network of the improved Real-ESRGAN model; L1 represents the mean absolute error loss; L percep Indicates perceived loss; L G represents the GAN network loss; η represents the coefficient of mean absolute error loss; λ represents the coefficient of GAN network loss.

[0021] Preferably, the perceptual loss L percep The expression is:

[0022] L percep =||p(x i )-φ(y i )||1;

[0023] Among them, ||||1 means to find the L1 norm; φ means the VGG19 model; x i Represents the i-th feature layer data input to the VGG19 model; y i Represents x i The corresponding real data.

[0024] Preferably, the feature layer data input to the VGG19 model includes data of the feature layer conv1_2 before activation, data of the feature layer conv2_2 before activation, data of the feature layer conv3_4 before activation, data of the feature layer conv4_4 before activation, and data of the feature layer conv5_4 before activation.

[0025] Preferably, the GAN network loss L G The expression is:

[0026]

[0027] Wherein, D represents the discriminant network of the improved Real-ESRGAN model; represents the true label; x i ' represents the data input to the discriminant network of the improved Real-ESRGAN model.

[0028] Preferably, the expression of the mean absolute error loss L1 is:

[0029] L1=mean||G(x i ")-y i "||1;

[0030] Wherein, mean represents the average value; ||||1 represents the L1 norm; G represents the generative network of the improved Real-ESRGAN model; x i " represents the data input to the discriminant network of the improved Real-ESRGAN model; y i " indicates x i "The corresponding real data.

[0031] Preferably, λ is 0.1.

[0032] Preferably, n is 1.

[0033] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a super-resolution image reconstruction method based on an improved Real-ESRGAN model, which can achieve the following beneficial technical effects:

[0034] 1) Improve the quality of reconstructed images: The self-attention mechanism can establish a direct connection between any two locations in the image, not just local areas. This enables the model to better capture the global dependencies in the image, thereby improving the quality of the generated image.

[0035] 2) Enhance the details of the reconstructed image: Since the self-attention mechanism can consider the information of all other positions in the image and obtain a larger receptive field and contextual information, it can help the model better restore and enhance details when generating high-resolution images.

[0036] 3) Improve model robustness: The self-attention mechanism is less dependent on the input features, so it can improve the robustness of the model. When there is noise or distortion in the input image, the self-attention mechanism can still capture the key information in the image and generate high-quality output images. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A structural diagram of a Z_RRDB provided in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the structure of an original Real-ESRGAN model provided in an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of a U-Net structure with spectral normalization provided in an embodiment of the present invention;

[0041] Figure 4 A comparison diagram of the effects of an original low-resolution image provided in a certain embodiment of the present invention and a reconstructed high-resolution image obtained by using the method of the present invention;

[0042] Figure 5 This is a comparison diagram of the effects of reconstructing high-resolution images obtained by using four algorithms respectively provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The following is an example of airport luggage image to illustrate the technical solution of the present invention.

[0045] The embodiment of the present invention discloses a super-resolution image reconstruction method based on an improved Real-ESRGAN model, comprising the following steps:

[0046] Input the low-resolution image of the airport baggage to be reconstructed into the generative network of the pre-trained improved Real-ESRGAN model to obtain the reconstructed high-resolution image of the airport baggage;

[0047] Among them, the generation network of the improved Real-ESRGAN model adds a self-attention module.

[0048] Further, such as Figure 1 and Figure 2 As shown, the generative network of the improved Real-ESRGAN model replaces all RRDB in the original Real-ESRGAN model generative network with Z_RRDB;

[0049] The Z_RRDB includes a first Conv_LReLU layer, a second Conv_LReLU layer, a third Conv_LReLU layer, a fourth Conv_LReLU layer, a fifth convolutional layer, a self-attention module, a sixth convolutional layer, an element-by-element addition unit, and a seventh convolutional layer connected in sequence;

[0050] The input end of the first Conv_LReLU layer is connected to the input ends of the second Conv_LReLU layer, the third Conv_LReLU layer, the fourth Conv_LReLU layer, and the fifth convolutional layer respectively;

[0051] The input end of the second Conv_LReLU layer is connected to the input ends of the third Conv_LReLU layer, the fourth Conv_LReLU layer, and the fifth convolutional layer respectively;

[0052] The input end of the third Conv_LReLU layer is connected to the input ends of the fourth Conv_LReLU layer and the fifth convolutional layer respectively;

[0053] The output of the fifth convolutional layer is connected to the input of the sixth convolutional layer;

[0054] The input of the first Conv_LReLU layer is connected to the input of the element-wise addition unit.

[0055] Further, such as Figure 3 As shown, the discriminant network of the improved Real-ESRGAN model adopts a U-Net structure with spectral normalization (the discriminant network of the original Real-ESRGAN model also adopts this structure).

[0056] It can be understood that U-Net consists of two parts: Encoder and Decoder. The encoder gradually reduces the resolution of the input image and captures high-level semantic information, while the decoder gradually restores the resolution of the image and outputs the final result. The U-Net architecture can effectively capture the features of the image at different scales, thereby providing richer information for judging the authenticity of the input image.

[0057] It can be understood that spectral normalization is a weight normalization technique, which can effectively control the weight size in the improved Real-ESRGAN model, prevent gradient explosion or vanishing problems during training, and thus enhance the stability of the model. The output of the discriminant network of the improved Real-ESRGAN model of the present invention is the authenticity value of each pixel, which enables it to provide detailed pixel-by-pixel feedback to the generative network. This feedback mechanism helps the generative network to more accurately capture the details and textures of the image during training, thereby generating more realistic high-resolution images. That is, the U-Net structure with spectral normalization not only ensures the overall authenticity of the generated high-resolution HR images, but also pays special attention to the retention of image details.

[0058] Furthermore, the total loss function of the generative network of the improved Real-ESRGAN model is:

[0059] L=L percep +λL G +ηL1;

[0060] Wherein, L represents the total loss of the generative network of the improved Real-ESRGAN model; L1 represents the mean absolute error loss; L percep Indicates perceived loss; L G represents the GAN network loss; η represents the coefficient of mean absolute error loss; λ represents the coefficient of GAN network loss.

[0061] Furthermore, the perceptual loss L percep The expression is:

[0062] L percep =||φ(x i )-φ(y i )||1;

[0063] Among them, ||| | represents the L1 norm (specifically, the sum of the absolute values ​​of the non-zero elements in the vector); φ represents the VGG19 model; x i Represents the i-th feature layer data input to the VGG19 model; y i Represents x i The corresponding real data.

[0064] Furthermore, the feature layer data input to the VGG19 model includes data of the feature layer conv1_2 before activation, data of the feature layer conv2_2 before activation, data of the feature layer conv3_4 before activation, data of the feature layer conv4_4 before activation, and data of the feature layer conv5_4 before activation.

[0065] Furthermore, the GAN network loss L G The expression is:

[0066]

[0067] Wherein, D represents the discriminant network of the improved Real-ESRGAN model; represents the true label; x i ' represents the data input to the discriminant network of the improved Real-ESRGAN model.

[0068] Furthermore, the expression of the mean absolute error loss L1 is:

[0069] L1=meam||G(x i ")-y i "||1;

[0070] Wherein, mean represents the average value; ||||1 represents the L1 norm (specifically, the sum of the absolute values ​​of the non-zero elements in the vector); G represents the generative network of the improved Real-ESRGAN model; x i " represents the data input to the discriminant network of the improved Real-ESRGAN model; y i " indicates x i "The corresponding real data.

[0071] Furthermore, λ is 0.1.

[0072] Furthermore, η is 1.

[0073] In order to verify the effectiveness of the improved Real-ESRGAN algorithm of the present invention, the present invention is compared with three algorithms: the SRGAN algorithm, the ESRGAN algorithm and the Real-ESRGAN algorithm.

[0074] It can be understood that the training method of the improved Real-ESRGAN model of the present invention is consistent with the training methods of the SRGAN algorithm, the ESRGAN algorithm, and the Real-ESRGAN algorithm;

[0075] Specifically, the four algorithms all selected the same 2500 images as the training set. During the training, the four algorithms all used the Adam optimizer, the learning rate was 0.0001, the training iteration batch size was 8, and 120 training times were performed. The training weights were saved every 10 training times, and the best weights were taken as the final weight values.

[0076] As shown in Table 1 and Table 2, the present invention uses PSNR (Peak Signal-to-Noise Ratio) and SSIM (structural similarity index measurement) as evaluation indicators to compare the four algorithms:

[0077] Table 1 Comparison of PSNR of super-resolution reconstructed images by four algorithms (unit: dB)

[0078]

[0079]

[0080] Table 2 Comparison of SSIM of super-resolution images reconstructed by four algorithms

[0081]

[0082] It can be seen from the data in Table 1 and Table 2 that the improved Real-ESRGAN of the present invention performs optimally in terms of both PSNR and SSIM.

[0083] In order to more intuitively experience the visual effect of the improved Real-ESRGAN algorithm of the present invention, Figure 4 A comparison diagram of the effects of the original low-resolution image and the reconstructed high-resolution image obtained by the method of the present invention is shown; Figure 4 The upper row of images are original low-resolution images, and the lower row are reconstructed high-resolution images obtained by the method of the present invention. It is not difficult to see that the quality of the lower row of images is obviously better than that of the upper row.

[0084] In order to verify that the present invention can better restore and enhance image details, the present invention partially enlarges the high-resolution luggage image reconstructed by the above four algorithms, such as Figure 5As shown in the figure, from left to right are the reconstructed high-resolution luggage images obtained by the SRGAN algorithm, the ESRGAN algorithm, the Real-ESRGAN algorithm and the improved Real-ESRGAN algorithm of the present invention. Obviously, the image reconstructed by the method of the present invention is clearer and has richer details.

[0085] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0086] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A super-resolution image reconstruction method based on the improved Real-ESRGAN model, It is characterized in that The following steps are involved: Input the low-resolution image to be reconstructed into the generative network of the pre-trained improved Real-ESRGAN model to obtain the reconstructed high-resolution image; Among them, the generation network of the improved Real-ESRGAN model adds a self-attention module.

2. According to claim 1, a super-resolution image reconstruction method based on an improved Real-ESRGAN model, It is characterized in that The generative network of the improved Real-ESRGAN model replaces all RRDBs in the original Real-ESRGAN model generative network with Z_RRDB; The Z_RRDB includes a first Conv_LReLU layer, a second Conv_LReLU layer, a third Conv_LReLU layer, a fourth Conv_LReLU layer, a fifth convolutional layer, a self-attention module, a sixth convolutional layer, an element-by-element addition unit, and a seventh convolutional layer connected in sequence; The input end of the first Conv_LReLU layer is connected to the input ends of the second Conv_LReLU layer, the third Conv_LReLU layer, the fourth Conv_LReLU layer, and the fifth convolutional layer respectively; The input end of the second Conv_LReLU layer is connected to the input ends of the third Conv_LReLU layer, the fourth Conv_LReLU layer, and the fifth convolutional layer respectively; The input end of the third Conv_LReLU layer is connected to the input ends of the fourth Conv_LReLU layer and the fifth convolutional layer respectively; The output of the fifth convolutional layer is connected to the input of the sixth convolutional layer; The input of the first Conv_LReLU layer is connected to the input of the element-wise addition unit.

3. The super-resolution image reconstruction method based on the improved Real-ESRGAN model according to claim 1, It is characterized in that The discriminant network of the improved Real-ESRGAN model adopts a U-Net structure with spectral normalization.

4. A super-resolution image reconstruction method based on an improved Real-ESRGAN model according to any one of claims 1 to 3, It is characterized in that The total loss function of the generative network of the improved Real-ESRGAN model is: L=L percep +λL G +ηL 1 ; Wherein, L represents the total loss of the generative network of the improved Real-ESRGAN model; L 1 represents the mean absolute error loss; L percep Indicates perceived loss; L G represents the GAN network loss; η represents the coefficient of mean absolute error loss; λ represents the coefficient of GAN network loss.

5. The super-resolution image reconstruction method based on the improved Real-ESRGAN model according to claim 4, It is characterized in that Perceptual loss L percep The expression is: L percep =||φ(x i )-φ(y i )|| 1 ; Among them, || || 1 represents the L1 norm; φ represents the VGG19 model; x i Represents the i-th feature layer data input to the VGG19 model; y i Represents x i The corresponding real data.

6. The super-resolution image reconstruction method based on the improved Real-ESRGAN model according to claim 5, It is characterized in that The feature layer data input to the VGG19 model includes data of the feature layer conv1_2 before activation, data of the feature layer conv2_2 before activation, data of the feature layer conv3_4 before activation, data of the feature layer conv4_4 before activation, and data of the feature layer conv5_4 before activation.

7. The super-resolution image reconstruction method based on the improved Real-ESRGAN model according to claim 4, It is characterized in that The GAN network loss L G The expression is: Wherein, D represents the discriminant network of the improved Real-ESRGAN model; represents the true label; x i ' represents the data input to the discriminant network of the improved Real-ESRGAN model.

8. The super-resolution image reconstruction method based on the improved Real-ESRGAN model according to claim 4, It is characterized in that The mean absolute error loss L 1 The expression is: L 1 =mean||G(x i ”)-y i ”|| 1 ; Among them, mean means finding the average value; || || 1 represents the L1 norm; G represents the generation network of the improved Real-ESRGAN model; x i " represents the data input to the discriminant network of the improved Real-ESRGAN model; y i " indicates x i "The corresponding real data.

9. The super-resolution image reconstruction method based on the improved Real-ESRGAN model according to claim 4, It is characterized in that λ is 0.

1.

10. The super-resolution image reconstruction method based on the improved Real-ESRGAN model according to claim 4, It is characterized in that η is 1.