Image super-resolution method based on auxiliary edge

Abstract

The present invention discloses an image super-resolution method based on auxiliary edges. The method includes an edge reconstruction subnetwork, an image reconstruction subnetwork, and an edge fusion module based on a pixel attention mechanism. The main steps are: (1) selecting a public image data set and downsampling to obtain a low-resolution image; (2) using an edge extraction algorithm to calculate the edge corresponding to the high-resolution image; (3) the image reconstruction subnetwork extracts features from the low-resolution image and reconstructs an initial high-resolution image; (4) the edge reconstruction subnetwork uses the low-level features of the image reconstruction subnetwork as input to reconstruct the initial edge image; (5) the edge fusion module uses the initial high-resolution image and the edge image as input, fuses the edge information, optimizes the initial high-resolution image, and obtains the final high-resolution image. The present invention can efficiently utilize edge information, suppress blur at the edge, and achieve high-quality super-resolution reconstruction.

Description

Technical Field

[0001] The invention belongs to the field of image super-resolution in computer vision, and in particular relates to an image super-resolution method based on auxiliary edges. Background Art

[0002] Among all the parameters of an image, resolution has a crucial impact on the quality of the image. Images with relatively low resolution contain less data, and a large amount of interesting details will be lost in the image, resulting in a poor visual experience. The most direct way to obtain high-resolution images is to use advanced hardware devices such as high-resolution cameras. However, these advanced high-resolution cameras are often expensive, making the system implementation cost high. In addition, there are physical limitations to improving the resolution of images from the perspective of replacing advanced hardware devices. It is not flexible enough and difficult to meet the requirements of different resolutions. Image super-resolution technology can reconstruct the corresponding high-resolution image from the degraded low-resolution image without changing the existing hardware. Compared with directly improving the hardware equipment, it is more flexible and convenient. In addition, improving the hardware equipment can only ensure a higher resolution of the currently acquired image. For existing low-resolution images, it is impossible to improve the resolution through a high-resolution camera.

[0003] However, many image super-resolution reconstruction methods directly consider the reconstruction problem of high-resolution images from the overall perspective. In the process, the regional features of specific pixels are not actually considered, such as whether the pixel belongs to the edge. The indiscriminate reconstruction of all pixels in the image is likely to cause the edges to be too smooth. Some methods explicitly reconstruct edge information from the perspective of the spatial domain and enhance the image edges. These methods that use image edges either directly build a parallel edge extraction network to construct the edge map corresponding to the high-resolution image from the low-resolution image, or obtain edge features from the reconstructed super-resolution image. These methods ignore the low-level features extracted in the image super-resolution reconstruction link, which must overlap with the low-level features extracted in the edge extraction task. In other words, these methods do not effectively utilize the feature information in the super-resolution reconstruction network, resulting in a certain degree of resource waste. Secondly, in these image super-resolution networks that explicitly use edge information, there is a feature fusion module for fusing the reconstructed high-resolution image and the edge image. In the design of this part, some methods cascade residual structures to form a more complex post-processing module, and some methods simply perform addition operations and cannot perform fusion efficiently. Summary of the invention

[0004] In order to solve the above problems, the present invention adopts the following technical solutions:

[0005] An image super-resolution method based on auxiliary edges comprises the following steps:

[0006] S1. Select a public image dataset as the true value, and use a bilinear interpolation algorithm to downsample the high-resolution image to obtain the corresponding low-resolution image;

[0007] S2. Calculate the corresponding edge image using an edge extraction algorithm based on the high-resolution image obtained in step S1;

[0008] S3. randomly select an area of ​​H×W in the low-resolution image in step S1 for cropping, and crop an image block of size sH×sW on the corresponding high-resolution image and the edge image in step S2, where s represents a scaling factor;

[0009] S4. Match the low-resolution image blocks obtained by cropping in step S3 with the high-resolution image blocks and edge image blocks to obtain a training data set;

[0010] S5. The image reconstruction subnetwork extracts features from low-resolution image patches and reconstructs the initial high-resolution image SR;

[0011] S6. Design of edge reconstruction subnetwork

[0012] 6.1: Design edge reconstruction module

[0013] The edge reconstruction module consists of multiple standard convolutional layers and deconvolutional layers in series. Each standard convolution is followed by a Relu activation function to improve the nonlinear fitting ability of the network. The number of deconvolutional layers N is dynamically determined according to the scaling factor, and the calculation formula is:

[0014]

[0015] 6.2: Design edge processing module

[0016] The edge processing module first combines the outputs of each edge reconstruction module using a splicing operation, and then uses a 1×1 standard convolution to reduce the dimension to obtain an edge image containing only one channel.

[0017] S7: Designing edge fusion module

[0018] 7.1: Designing an enhancement module based on pixel attention mechanism

[0019] This module contains three pixel attention modules. Each pixel attention module contains three branches. The first branch uses a 1×1 convolution kernel to compress the input features, and then uses the Sigmoid function to calculate the weight value W of each spatial position. The second branch uses a 3×3 convolution kernel to extract features from the input data, and the output result is multiplied by the output data in the first branch. The third branch directly outputs the input data and adds it to the output result of the second branch.

[0020] 7.2: Design edge fusion module

[0021] The edge fusion module first sets up two standard convolutions with a convolution step of 2. After each standard convolution, an enhancement module based on the pixel attention mechanism is set. Then two deconvolutions are used to restore the spatial resolution of the features, and an enhancement module based on the pixel attention mechanism is set before each deconvolution. The output of the first deconvolution is summed with the output of the first standard convolution again as the input of the next convolution.

[0022] S8. Set the loss function and train the network to obtain the mapping model.

[0023] As a further improvement of the present invention, in the auxiliary edge-based image super-resolution method, in step S1, Gaussian noise is first added to the high-resolution image, and then down-sampled to simulate the actual environment.

[0024] As a further improvement of the present invention, in the auxiliary edge-based image super-resolution method, the high-resolution image and the corresponding edge image are used as true values ​​in step S2.

[0025] As a further improvement of the present invention, in the auxiliary edge-based image super-resolution method, in step S5, only the image reconstruction subnetwork extracts features from the input image, and the edge reconstruction subnetwork takes some low-level features of the image reconstruction subnetwork as input.

[0026] As a further improvement of the present invention, in the auxiliary edge-based image super-resolution method, in step 6.1, the convolution kernel size of the standard convolution is 3×3, and the convolution step size is 1. The number of deconvolutions is dynamically adjusted according to the scaling factor.

[0027] As a further improvement of the present invention, in the auxiliary edge-based image super-resolution method, in step 7.2, each convolution is connected in series with an enhancement module based on a pixel attention mechanism, and each enhancement module based on a pixel attention mechanism contains three serially connected pixel attention modules.

[0028] As a further improvement of the present invention, in the auxiliary edge-based image super-resolution method, in step S8, the loss function is jointly composed of the l1 loss function, the Charbonnier loss function and the edge attention loss function, and the weight values ​​are 0.2, 1, and 0.5 respectively.

[0029] Beneficial effects:

[0030] The present invention aims at the problems that low-level features in the existing image super-resolution network based on explicit edges cannot be fully utilized, the network efficiency is low, and the reconstructed edges cannot be efficiently fused. An image super-resolution method based on auxiliary edges is proposed. The present invention explicitly utilizes edge information to enhance the image edge. The image super-resolution reconstruction is divided into three parts: an image super-resolution reconstruction subnetwork, an edge reconstruction subnetwork, and an edge fusion module. In the edge reconstruction subnetwork, the input data comes from the image reconstruction subnetwork, avoiding the problem that a simple parallel structure is difficult to effectively utilize low-level features and a serial structure reconstructs edge maps from high-resolution images. For the fusion of edge information and high-resolution images, the present invention combines the relevant knowledge of image deblurring and designs a codec optimization module combined with a pixel attention mechanism to improve the final reconstruction quality. In network training, the present invention uses a joint loss function based on the l1 loss function, the Charbonnier loss function, and the edge attention loss function to emphasize the spatial characteristics of edge pixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the framework of the entire system of the invention.

[0032] Figure 2 It is a schematic diagram of the edge reconstruction sub-network.

[0033] Figure 3 It is a schematic diagram of the edge fusion module.

[0034] Figure 4 These are some high-resolution images and their corresponding edge images, where the left column of images are high-resolution images and the right column of images are edge images corresponding to the left column of images.

[0035] Figure 5 This is the result of enlarging some images in the test set four times.

[0036] Figure 6 These are the various indicators (PSNR / SSIM) of the invented method on the test set. DETAILED DESCRIPTION

[0037] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. Pytorch is selected as the deep learning framework for network implementation under the Linux operating system. The hardware environment is Nvidia RTX3070 and Intel (R) i510600KF CPU. The deep learning framework and the hardware environment are only for implementing and testing the proposed method to confirm the effectiveness of the super-resolution reconstruction method proposed in this patent. It should be understood that these examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, modifications to various equivalent forms of the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0038] An image super-resolution network method based on explicit edges, the system framework is shown in the figure Figure 1 shown.

[0039] The specific steps include:

[0040] Step 1: Select the public datasets DIV2K and Flickr2K as training datasets, and select Set5, Set14, B100, and Urban100 as test datasets. The images in these image datasets are used as high-resolution images, and the bilinear interpolation algorithm is used to downsample the images in the dataset to obtain the corresponding low-resolution images, with a downsampling factor of 1 / s;

[0041] Step 2: First, perform Gaussian filtering on the high-resolution image to remove some noise points, and the Gaussian kernel size is set to 3. Then convert the filtered high-resolution image into a grayscale image, select the soft edge calculation method, and use the grayscale image to calculate the low-resolution image corresponding to the high-resolution image in step 1;

[0042] Step 3: Randomly select an area of ​​H×W in the low-resolution image in step 1 for cropping, and crop an image block of size sH×sW on the corresponding high-resolution image and the edge image in step 2, where s represents the scaling factor;

[0043] Step 4: Randomly rotate the obtained image blocks by 90°, 180°, and 270° to enhance the dataset;

[0044] Step 5: Design an auxiliary edge-based image super-resolution network

[0045] 5.1: Image Super-Resolution Reconstruction Sub-Network

[0046] A publicly published image super-resolution reconstruction algorithm and an image super-resolution network based on multi-scale feature aggregation are selected.

[0047] 5.2: Edge reconstruction sub-network

[0048] like Figure 2, the edge reconstruction subnetwork consists of multiple edge reconstruction modules and an edge processing module. The edge reconstruction module consists of two standard convolutional layers, a Relu function, and a dynamically adjusted number of deconvolutional layers in series. The edge reconstruction module takes the low-level features of different stages of the image super-resolution reconstruction subnetwork as input and outputs a high-resolution edge image. The edge processing module first splices the edge images output by each edge reconstruction module, and then compresses the spliced ​​features using 1×1 convolution to obtain the final edge image. The number of channels of the features in the edge reconstruction subnetwork is set to 32, and the output features of the first 6 layers of the image super-resolution reconstruction subnetwork are selected as the input of the edge reconstruction module, and all layers less than 6 are selected.

[0049] Step 6: Design the edge blending module

[0050] like Figure 3 , the edge fusion module first sets two standard convolutions with a convolution step of 2. After each standard convolution, an enhancement module based on the pixel attention mechanism is set. Then two deconvolutions are used to restore the spatial resolution of the feature, and an enhancement module based on the pixel attention mechanism is set before each deconvolution. The output of the first deconvolution is summed with the output of the first standard convolution again as the input of the next convolution. The high-resolution image output by the image super-resolution reconstruction subnetwork is summed with the output of the edge fusion module as the main body to generate the final high-resolution image. The spatial resolution of the feature in the edge fusion module decreases by 1 / 2 after each layer of standard convolution, and the number of feature channels increases by 2 times; after each layer of deconvolution, the spatial resolution of the feature increases by 2 times after each layer of standard convolution, and the number of feature channels decreases by 1 / 2.

[0051] Step 7: Train the auxiliary edge-based image super-resolution network

[0052] 7.1 Setting the loss function

[0053] The loss function is composed of the l1 loss function, the Charbonnier loss function and the edge attention loss function, with weight values ​​of 0.2, 1, and 0.5 respectively. The various parameters in the network are calculated by minimizing the loss between the reconstructed high-resolution image and the real high-resolution image.

[0054] 7.2 Setting training parameters

[0055] The number of training rounds is set to 500, and in each round, 16,000 cropped small images are randomly selected from the dataset as the input of the network model. The initial learning rate of the network is set to 0.0001, and it is halved after 200 rounds. As the number of training rounds increases, the learning rate decreases faster. The network is iteratively trained using the Adam optimizer, and the optimizer parameters are set to β1 = 0.9, β2 = 0.999 and ε = 10 -8 .

[0056] 7.3 Evaluation Criteria

[0057] First, the reconstructed high-resolution image is converted from RGB space to YCbCr space, and then the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) values ​​are calculated only on the Y channel.

[0058] It should be noted that the above implementation examples are only cases made for clear explanation, and are not intended to limit the implementation methods. It is not necessary and impossible to list all the implementation methods here. All components not specified in this embodiment can be implemented using existing technologies. For those of ordinary skill in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. An image super-resolution method based on auxiliary edges, characterized in that: The following steps are involved: S1. Select a public image dataset as the true value, and use a bilinear interpolation algorithm to downsample the high-resolution image to obtain the corresponding low-resolution image; S2. Calculate the corresponding edge image using an edge extraction algorithm based on the high-resolution image obtained in step S1; S3. randomly select an area of ​​H×W in the low-resolution image in step S1 for cropping, and crop an image block of size sH×sW on the corresponding high-resolution image and the edge image in step S2, where s represents a scaling factor; S4. Match the low-resolution image blocks obtained by cropping in step S3 with the high-resolution image blocks and edge image blocks to obtain a training data set; S5. The image reconstruction subnetwork extracts features from low-resolution image patches and reconstructs the initial high-resolution image SR; S6. Design of edge reconstruction subnetwork 6.1: Design edge reconstruction module The edge reconstruction module consists of multiple standard convolutional layers and deconvolutional layers in series. Each standard convolution is followed by a Relu activation function to improve the nonlinear fitting ability of the network. The number of deconvolutional layers N is dynamically determined according to the scaling factor, and the calculation formula is: 6.2: Design edge processing module The edge processing module first combines the outputs of each edge reconstruction module using a splicing operation, and then uses a 1×1 standard convolution to reduce the dimension to obtain an edge image containing only one channel; S7: Designing edge fusion module 7.1: Designing an enhancement module based on pixel attention mechanism This module contains three pixel attention modules; Each pixel attention module contains three branches; The first branch uses a 1×1 convolution kernel to compress the input features, and then uses the Sigmoid function to calculate the weight value W of each spatial position; the second branch uses a 3×3 convolution kernel to extract features from the input data, and the output result is multiplied by the output data in the first branch; the third branch directly outputs the input data and adds it to the output result of the second branch; 7.2: Design edge fusion module The edge fusion module first sets two standard convolutions with a convolution step of 2; after each standard convolution, an enhancement module based on the pixel attention mechanism is set; then two deconvolutions are used to restore the spatial resolution of the features, and before each deconvolution, an enhancement module based on the pixel attention mechanism is set; the output of the first deconvolution is summed with the output of the first standard convolution again as the input of the next convolution; S8. Set the loss function and train the network to obtain the mapping model.

2. The image super-resolution method based on auxiliary edges according to claim 1, characterized in that: In step S1, Gaussian noise is first added to the high-resolution image, and then down-sampled to simulate the actual environment.

3. The image super-resolution method based on auxiliary edges according to claim 1, characterized in that: In step S2, the high-resolution image and the corresponding edge image are used as the true value.

4. The image super-resolution method based on auxiliary edge according to claim 1, characterized in that: In step S5, only the image reconstruction subnetwork extracts features from the input image, and the edge reconstruction subnetwork takes some low-level features of the image reconstruction subnetwork as input.

5. The image super-resolution method based on auxiliary edges according to claim 1, characterized in that: In step 6.1, the convolution kernel size of the standard convolution is 3×3, and the convolution step size is 1; the number of deconvolutions is dynamically adjusted according to the scaling factor.

6. The image super-resolution method based on auxiliary edges according to claim 1, characterized in that: In step 7.2, each convolution is connected in series with an enhancement module based on the pixel attention mechanism, and each enhancement module based on the pixel attention mechanism contains three serial pixel attention modules.

7. The image super-resolution method based on auxiliary edges according to claim 1, characterized in that: In step S8, the loss function is composed of the l1 loss function, the Charbonnier loss function and the edge attention loss function, and the weight values ​​are 0.2, 1, and 0.5 respectively.

Images