A super-resolution reconstruction method based on SRCNN supplementary module

Abstract

The present invention discloses a super-resolution reconstruction method based on an SRCNN supplementary module, belonging to the field of computer vision technology. The method comprises: preprocessing a data set to obtain a low-resolution image using a bicubic linear interpolation method; partitioning the low-resolution image and the corresponding original high-resolution image into blocks; taking the brightness channel of the block-processed image blocks as input data for a reconstruction model; constructing a reconstruction model, initializing model parameters, and determining a mean square error as a loss function; inputting low-resolution images from a training set to obtain a reconstructed image, updating the model parameters using an Adam optimizer, and obtaining a trained reconstruction model through iterative calculation; and inputting the low-resolution image into the trained reconstruction model to obtain a high-resolution image. The present invention can supplement the model with information that has been set to zero, thereby improving the performance of the model. After three layers of convolution operations, a reconstructed high-resolution image can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a super-resolution reconstruction method based on an SRCNN supplementary module. Background Art

[0002] With the advancement of technology, transmitting high-resolution images has become possible, and our requirements for image and video quality, specifically resolution, are constantly increasing. However, obtaining high-resolution images requires advanced equipment and a good shooting environment. Since higher resolutions take up more space, they also require a large amount of storage space. Generally speaking, achieving these three requirements is not easy. As a result, images often appear blurry, with low resolution and unclear details. This not only affects the visual experience but also makes it difficult to accurately identify objects in professional applications such as surveillance, facial recognition, and medicine.

[0003] Due to the aforementioned reasons, improving image resolution has become a hot topic of research. The hardware solution involves using higher-resolution equipment. However, high-resolution equipment is not only expensive but also faces manufacturing limitations, making it impractical for widespread adoption. Therefore, researchers are beginning to address this issue from an algorithmic perspective. Image super-resolution reconstruction algorithms have been developed for decades since their introduction and have achieved impressive results. Traditional super-resolution reconstruction algorithms are categorized as interpolation-based, degradation model-based, and learning-based. Super-resolution algorithms based on deep learning are currently a hot topic of research.

[0004] Deep learning extracts data features and learns patterns through multi-layer nonlinear transformations, making predictions about new data in the process. Image super-resolution reconstruction algorithms based on deep learning significantly outperform traditional super-resolution reconstruction algorithms in terms of image detail and speed. Existing deep learning super-resolution reconstruction algorithms can be divided into two categories based on whether a training dataset is required: supervised learning, which requires a training dataset; and unsupervised learning, such as zero-shot super-resolution, which uses information from the image itself, different scaling factors, and various enhancement methods to create a dataset for cropping, rotating, and flipping images. There are also deep image prior algorithms, which design a generator network, input a low-resolution image, and iteratively optimize the generator to generate a high-resolution image.

[0005] SRCNN networks use the Relu (Reinforced Lu) function as the activation function between network layers. The Relu activation function is often used due to its fast training speed and ability to address the vanishing gradient problem in deep learning. However, the Relu activation function sets negative inputs to zero, which directly deactivates some neurons and loses some data information. For super-resolution reconstruction algorithms, overfitting is rarely a problem, so in super-resolution environments, it is necessary to consider reintegrating the lost information into the model. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a super-resolution reconstruction method based on the SRCNN supplementary module, which can supplement the information set to 0 into the model, improve the performance of the model, and obtain a reconstructed high-resolution image after three-layer convolution operation.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A super-resolution reconstruction method based on an SRCNN supplementary module comprises the following steps:

[0009] Step 1: Preprocess the dataset and obtain a low-resolution image using bicubic linear interpolation.

[0010] Step 2: Divide the acquired low-resolution image and the corresponding original high-resolution image into blocks;

[0011] Step 3: Take the brightness channel of the image block after block processing as the input data of the reconstruction model;

[0012] Step 4: Construct a super-resolution reconstruction model based on the SRCNN supplementary module. The Relu function is used as the activation function of the reconstruction model. The rRelu function extracts the negative value information set to 0 and passes it into the SRCNN supplementary module. The reconstruction model parameters are initialized and the mean square error is determined as the loss function.

[0013] Step 5: Input the low-resolution image in the training set to obtain the reconstructed image, compare it with the original high-resolution image, and use the Adam optimizer to update the model parameters to minimize the difference between the reconstructed image and the original high-resolution image. After iterative calculation, the trained reconstruction model is obtained.

[0014] Step 6: Input the low-resolution image to be reconstructed into the trained reconstruction model to obtain the corresponding high-resolution image.

[0015] A further improvement of the technical solution of the present invention is that in step 3, the method for converting the image brightness channel from the RGB image is:

[0016] Y=0.257*R+0.504*G+0.098*B+16

[0017] Among them, Y represents the brightness channel, R represents the red channel, G represents the green channel, and B represents the blue channel.

[0018] A further improvement of the technical solution of the present invention is that: in step 4, the SRCNN supplementary module is located after the network convolution layer, and the rRelu function is located before the convolution layer in the SRCNN supplementary module. The SRCNN supplementary module is composed of the rRelu function and the 1*1 convolution layer.

[0019] A further improvement of the technical solution of the present invention is that in step 4, the analytical expression of the ReLU function is:

[0020] Relu=max(0,x)

[0021] The analytical expression of the rRelu function is:

[0022] rRelu=min(0,x).

[0023] A further improvement of the technical solution of the present invention is that in step 4, the expression using the mean square error as the loss function is:

[0024]

[0025] Where n is the number of training samples, X i is a high-resolution image, Y i is a low-resolution image, θ is a network parameter, F(Y i ,θ) is the reconstructed image.

[0026] A further improvement of the technical solution of the present invention is that in step 5, the weights are updated during the training process by the following method:

[0027]

[0028] Among them, Δ represents the step size, l represents the number of convolutional layers, w is the weight parameter in the network, i is the index value, and η is the learning rate in the model. is the derivative.

[0029] A further improvement of the technical solution of the present invention is that the super-resolution reconstruction model based on the SRCNN supplementary module includes:

[0030] The first convolutional layer is used to extract image blocks, which are represented as high-dimensional vectors, including feature maps. The number of feature maps is equal to the appropriate number of dimensions.

[0031] The second convolution layer is used to nonlinearly map the high-dimensional vector to another high-dimensional vector. It includes another set of feature maps, which are the expression of high-resolution image blocks. This convolution layer is called nonlinear mapping.

[0032] The third convolution layer is used to reconstruct the image blocks and synthesize the high-resolution features obtained from the above two layers into a complete image;

[0033] Filter weight and parameter optimization process:

[0034] The first layer operation is expressed as:

[0035] F0(Y)=max(0,W0*Y+B0)

[0036] The second layer operation is expressed as:

[0037] F1(Y)=max(0,W1*F0(Y)+B1)

[0038] The third layer operation is expressed as:

[0039] F2(Y)=W2*F1(Y)+B2

[0040] Among them, W represents the filter, B represents the product, and * represents the convolution operation.

[0041] A further improvement of the technical solution of the present invention is that the reconstruction model uses the Adam optimizer to update the parameters, and in the three-layer network structure of the reconstruction model, the learning rate of the first two convolutional layers is 10 -4 , the third layer learning rate is 10 -5 .

[0042] A further improvement of the technical solution of the present invention is that the SRCNN supplementary module uses the rRelu function to extract information, reduces the dimension of the inactivated negative information through the convolution kernel, and re-inputs it into the model, which is expressed as:

[0043] S(Y)=min(0,W*Y+B)

[0044] Among them, s(Y) represents the convolution operation of the supplementary module, W corresponds to n c*1*1 convolution kernels, c is the number of channels of the input image, the size of each convolution kernel is c*1*1, and the output consists of n feature maps.

[0045] Due to the adoption of the above technical solution, the technical advancements achieved by the present invention are:

[0046] 1. The present invention achieves the technical effect of solving the neuron inactivation phenomenon caused by the ReLU activation function by proposing a super-resolution reconstruction method based on the SRCNN supplementary module. The super-resolution reconstruction method based on the SRCNN supplementary module is an end-to-end method for reconstructing high-resolution images.

[0047] 2. The present invention proposes to extract negative information and then add it back into the model, and adopts peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as evaluation indicators of image quality, thereby achieving the technical effect of improving model performance and increasing the numerical value of evaluation indicators.

[0048] 3. The present invention extracts negative value information through the rRelu function and re-adds it to the model, solving the problem that the Relu function cannot activate the input negative value, causing partial neuron inactivation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is the ReLU function image in the present invention;

[0050] Figure 2 is the reverse ReLU function image in the present invention;

[0051] Figure 3 It is a structural block diagram of the supplementary module in the present invention;

[0052] Figure 4 It is a flow chart of the super-resolution reconstruction method based on the SRCNN supplementary module in the present invention;

[0053] Figure 5 This is a network structure diagram of the super-resolution reconstruction method based on the SRCNN supplementary module in the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments:

[0055] The present invention is proposed to solve the problems existing in the Relu activation function in convolutional neural networks, such as Figure 1 As shown, when 0 to positive infinity, the output value of the Relu function is the same as the input value, but when negative infinity to 0, the output value is 0. In order to solve the problem of setting negative input values ​​to 0, the present invention proposes a reverse Relu (hereinafter referred to as rRelu) function, as shown in Figure 2 As shown in , when the input is negative, the output of the rRelu function is the same as the input value, and when the input is positive, the output value is 0. Figure 3 As can be seen from the supplementary module block diagram, after the negative value information is extracted by the rRelu function, it is re-added to the model after the dimensionality reduction operation is performed through the 1*1 convolution. The input low-resolution image is reconstructed through the model, and then the reconstructed image is compared with the original high-resolution image. The network is optimized by the loss function MSE. After convergence, a network model that can reconstruct the low-resolution image into the corresponding high-resolution image is obtained.

[0056] like Figure 4 As shown, a super-resolution reconstruction method based on the SRCNN supplementary module includes the following steps:

[0057] Step 1: Preprocess the dataset and obtain a low-resolution image using bicubic linear interpolation.

[0058] Step 2: Divide the acquired low-resolution image and the corresponding original high-resolution image into blocks;

[0059] Step 3: Take the brightness channel of the image block after block processing as the model input data;

[0060] The method for converting the image brightness channel from RGB image is:

[0061] Y=0.257*R+0.504*G+0.098*B+16

[0062] Among them, Y represents the brightness channel, R represents the red channel, G represents the green channel, and B represents the blue channel.

[0063] Step 4: Construct a super-resolution reconstruction model based on the SRCNN supplementary module. The Relu function is used as the activation function of the reconstruction model. The rRelu function extracts the negative value information set to 0 and passes it into the SRCNN supplementary module. Initialize the model parameters and determine the mean square error as the loss function.

[0064] Among them, the SRCNN supplementary module is located after the network convolution layer, and the rRelu function is located before the convolution layer in the SRCNN supplementary module. The SRCNN supplementary module is composed of the rRelu function and the 1*1 convolution layer.

[0065] The analytical expression of the Relu function is:

[0066] Relu=max(0,x)

[0067] The analytical expression of the rRelu function is:

[0068] rRelu=min(0,x)

[0069] Use mean square error as the loss function:

[0070]

[0071] Where n is the number of training samples, X i is a high-resolution image, Y i is a low-resolution image, θ is a network parameter, F(Y i ,θ) is the reconstructed image.

[0072] Step 5: Input the low-resolution image in the training set to obtain the reconstructed image, compare it with the original high-resolution image, and use the Adam optimizer to update the model parameters to minimize the difference between the reconstructed image and the original high-resolution image. After iterative calculation, the trained reconstruction model is obtained.

[0073] During training, weights are updated using the following methods:

[0074]

[0075] Among them, Δ represents the step size, l represents the number of convolutional layers, w is the weight parameter in the network, i is the index value, and η is the learning rate in the model. is the derivative.

[0076] The super-resolution reconstruction model based on the SRCNN supplementary module uses the Adam optimizer to update the parameters. In the three-layer network structure of the super-resolution reconstruction model based on the SRCNN supplementary module, the learning rate of the first two convolutional layers is 10 -4 , the third layer learning rate is 10 -5 .

[0077] like Figure 5 As shown in Figure 1, the trained reconstruction model includes the first convolutional layer, the second convolutional layer, and the third convolutional layer connected sequentially.

[0078] The first convolutional layer extracts image patches, representing them as high-dimensional vectors, including feature maps. The number of feature maps is equal to the appropriate number of dimensions. The second convolutional layer nonlinearly maps the high-dimensional vectors onto another high-dimensional vector, including another set of feature maps. This represents the high-resolution image patch, and this convolution layer is called nonlinear mapping. The third convolutional layer reconstructs the image patch, combining the high-resolution features obtained in the previous two layers to form a complete image. In this model, all filter weights and parameters must be optimized.

[0079] The first layer operation is expressed as:

[0080] F0(Y)=max(0,W0*Y+B0)

[0081] The second layer operation is expressed as:

[0082] F1(Y)=max(0,W1*F0(Y)+B1)

[0083] The third layer operation is expressed as:

[0084] F2(Y)=W2*F1(Y)+B2

[0085] Among them, W represents the filter, B represents the product, and * represents the convolution operation.

[0086] The SRCNN supplementary module uses the rRelu function to extract information, reduces the dimension of the inactivated negative information through the convolution kernel, and re-inputs it into the model, which is expressed as:

[0087] S(Y)=min(0,W*Y+B)

[0088] The first-layer supplementary module is represented as:

[0089] r1(Y)=min(0,W0*Y+B0)

[0090]

[0091] The second-layer supplementary module is expressed as:

[0092] r2(Y)=min(0,W1*F0(Y)+B1)

[0093]

[0094] The third-layer supplementary module is represented as:

[0095] r3(Y)=min(0,W2*F1(Y)+B2)

[0096]

[0097] Among them, r(Y) represents the information extracted by the rRelu function, s(Y) represents the convolution operation of the supplementary module, in the s(Y) formula, W corresponds to n c*1*1 convolution kernels, c is the number of channels of the input image, the size of each convolution kernel is c*1*1, and the output consists of n feature maps.

[0098] Step 6: Input the low-resolution image to be reconstructed into the trained reconstruction model to obtain the corresponding high-resolution image.

[0099] Example

[0100] Due to the SRCNN network's simple structure, it doesn't require a large dataset for training to converge. Furthermore, considering factors such as video memory and runtime speed, this example uses the first 100 images from the DIV2K dataset as the training dataset. During image preprocessing, the images are sliced ​​into 33x33 blocks, and low-resolution images undergo bicubic linear interpolation. Finally, training image pairs corresponding to the low-resolution and high-resolution images are generated.

[0101] In the super-resolution reconstruction method model based on the SRCNN supplementary module, the first convolution layer is based on the convolution kernel f (9*9) size, the output channel is n (64), the number of convolution kernels is the same as the output channels, and in order to keep the output image size consistent, the padding value is 4. The convolution kernel size of the second convolution layer is 5*5, the output channel is 32, and the padding is 2. The convolution kernel size of the third convolution layer is 5*5, the output channel is consistent with the number of input image channels, and the padding is 2. In the supplementary module, 1*1 convolution is used for dimensionality reduction, reducing the dimension to 1, and then inputting it into the next convolution layer.

[0102] This example uses peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as evaluation indicators of model performance, as shown in Table 1:

[0103] Table 1 Comparison of SRCNN model and SRCNN model with supplementary module (SM-SRCNN)

[0104]

[0105]

[0106] From the comparison in Table 1, it can be seen that the performance of the SRCNN model with the addition of the supplementary module is better than that of the original SRCNN model. In the super-resolution model, the supplementary module can bring better results for reconstructing images.

[0107] In summary, the present invention can add information set to 0 to the model, which can improve the performance of the model. After three layers of convolution operations, a reconstructed high-resolution image can be obtained.

Claims

1. A super-resolution reconstruction method based on an SRCNN supplementary module, characterized by: The following steps are involved: Step 1: Preprocess the dataset and obtain a low-resolution image using bicubic linear interpolation. Step 2: Divide the acquired low-resolution image and the corresponding original high-resolution image into blocks; Step 3: Take the brightness channel of the image block after block processing as the input data of the reconstruction model; Step 4: Construct a super-resolution reconstruction model based on the SRCNN supplementary module. The Relu function is used as the activation function of the reconstruction model. The rRelu function extracts the negative value information set to 0 and passes it into the SRCNN supplementary module. The reconstruction model parameters are initialized and the mean square error is determined as the loss function. The SRCNN supplementary module is located after the network convolution layer, and the rRelu function is located before the convolution layer in the SRCNN supplementary module. The SRCNN supplementary module consists of the rRelu function and the 1*1 convolution layer; The analytical expression of the Relu function is: Relu=max(0,x) The analytical expression of the rRelu function is: rRelu=min(0,x) The super-resolution reconstruction model based on the SRCNN supplementary module includes: The first convolutional layer is used to extract image blocks, which are represented as high-dimensional vectors, including feature maps. The number of feature maps is equal to the appropriate number of dimensions. The second convolution layer is used to nonlinearly map the high-dimensional vector to another high-dimensional vector. It includes another set of feature maps, which are the expression of high-resolution image blocks. This convolution layer is called nonlinear mapping. The third convolution layer is used to reconstruct the image blocks and synthesize the high-resolution features obtained from the above two layers into a complete image; Filter weight and parameter optimization process: The first layer operation is expressed as: F0(Y)=max(0,W0*Y+B0) The second layer operation is expressed as: F1(Y)=max(0,W1*F0(Y)+B1) The third layer operation is expressed as: F2(Y)=W2*F1(Y)+B2 Among them, W represents the filter, B represents the product, and * represents the convolution operation; The reconstruction model uses the Adam optimizer to update the parameters. In the three-layer network structure of the reconstruction model, the learning rate of the first two convolutional layers is 10 -4 , the third layer learning rate is 10 -5 ; Step 5: Input the low-resolution image in the training set to obtain the reconstructed image, compare it with the original high-resolution image, and use the Adam optimizer to update the model parameters to minimize the difference between the reconstructed image and the original high-resolution image. After iterative calculation, the trained reconstruction model is obtained. Step 6: Input the low-resolution image to be reconstructed into the trained reconstruction model to obtain the corresponding high-resolution image.

2. The super-resolution reconstruction method based on the SRCNN supplementary module according to claim 1, characterized in that: In step 3, the image brightness channel is converted from the RGB image as follows: Y=0.257*R+0.504*G+0.098*B+16 Among them, Y represents the brightness channel, R represents the red channel, G represents the green channel, and B represents the blue channel.

3. The super-resolution reconstruction method based on the SRCNN supplementary module according to claim 1, characterized in that: In step 4, the expression using mean square error as the loss function is: Where n is the number of training samples, X i is a high-resolution image, Y i is a low-resolution image, θ is a network parameter, F(Y i ,θ) is the reconstructed image.

4. The super-resolution reconstruction method based on the SRCNN supplementary module according to claim 1, characterized in that: In step 5, the weights are updated during training by: Among them, Δ represents the step size, l represents the number of convolutional layers, w is the weight parameter in the network, i is the index value, and η is the learning rate in the model. is the derivative.

5. The super-resolution reconstruction method based on the SRCNN supplementary module according to claim 1, characterized in that: The SRCNN supplementary module uses the rRelu function to extract information, reduces the dimension of the inactivated negative information through the convolution kernel, and re-inputs it into the model, which is expressed as: S(Y)=min(0,W*Y+B) Among them, s(Y) represents the convolution operation of the supplementary module, W corresponds to n c*1*1 convolution kernels, c is the number of channels of the input image, the size of each convolution kernel is c*1*1, and the output consists of n feature maps.

Images