Image super-resolution reconstruction method and system based on residual channel attention network

Abstract

The present invention belongs to the field of image processing technology and provides an image super-resolution reconstruction method and system based on a residual channel attention network. A high-resolution reconstructed image is obtained based on a low-resolution image to be reconstructed and an image super-resolution reconstruction model. The construction process of the image super-resolution reconstruction model includes: shallow feature extraction and deep feature extraction, wherein shallow features are obtained through a shallow feature channel, a deep feature extraction model is constructed based on the residual channel attention network, and deep features are extracted based on the shallow features and the deep feature extraction model. The deep feature extraction model includes multiple pixel and channel attention networks, each of which has a channel attention unit, a pixel attention unit, and an Inception unit arranged in parallel inside, and a residual structure is added in parallel on the outer layer. This allows the network to extract more and more useful information, thereby achieving higher-precision super-resolution reconstruction.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular to an image super-resolution reconstruction method and system based on a residual channel attention network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] At present, image super-resolution methods can be mainly divided into the following three categories: interpolation-based super-resolution reconstruction, reconstruction-based super-resolution reconstruction, and learning-based super-resolution reconstruction.

[0004] Interpolation-based super-resolution reconstruction is the simplest method for improving resolution, but its reconstruction effect is not very good. Reconstruction-based super-resolution reconstruction technology has improved the problem of details, but its performance decreases as the scale factor increases. This method is also time-consuming. Among learning-based super-resolution reconstruction methods, deep learning, as the name suggests, has a very deep network, which can lead to problems such as gradient vanishing and overfitting. Summary of the Invention

[0005] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides an image super-resolution reconstruction method and system based on the residual channel attention network, which proposes an improved method of improving image resolution by using channel attention, while improving image resolution and avoiding problems such as gradient disappearance.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A first aspect of the present invention provides an image super-resolution reconstruction method based on a residual channel attention network, comprising the following steps:

[0008] Acquire a low-resolution image to be reconstructed;

[0009] A high-resolution reconstructed image is obtained based on a low-resolution image to be reconstructed and an image super-resolution reconstruction model; wherein the construction process of the image super-resolution reconstruction model includes: shallow feature extraction and deep feature extraction, wherein shallow features are obtained through a shallow feature channel, a deep feature extraction model is constructed based on a residual channel attention network, and deep features are extracted based on the shallow features and the deep feature extraction model. The deep feature extraction model includes multiple pixel and channel attention networks, and each pixel and channel attention network has a channel attention unit, a pixel attention unit, and an Inception unit arranged in parallel inside, and a residual structure is added in a parallel outer layer.

[0010] A second aspect of the present invention provides an image super-resolution reconstruction system based on a residual channel attention network, comprising:

[0011] The image acquisition module is configured to: acquire a low-resolution image to be reconstructed;

[0012] The high-resolution image reconstruction module is configured to obtain a high-resolution reconstructed image based on a low-resolution image to be reconstructed and an image super-resolution reconstruction model; wherein the construction process of the image super-resolution reconstruction model includes: shallow feature extraction and deep feature extraction, wherein the shallow feature is obtained through the shallow feature channel,

[0013] A deep feature extraction model is constructed based on the residual channel attention network, and deep features are extracted according to the shallow features and the deep feature extraction model. The deep feature extraction model includes multiple pixel and channel attention networks. Channel attention units, pixel attention units and Inception units are set in parallel inside each pixel and channel attention network, and a residual structure is added to the outer layer in parallel.

[0014] A third aspect of the present invention provides a computer-readable storage medium.

[0015] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the image super-resolution reconstruction method based on the residual channel attention network as described above.

[0016] A fourth aspect of the present invention provides a computer device.

[0017] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the image super-resolution reconstruction method based on the residual channel attention network as described above are implemented.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] High-precision image super-resolution is achieved by using a deep pixel and channel attention (PACA) network. The PACA network is deeper than previous convolutional neural network-based methods to achieve better reconstruction effects. Residual learning is used to enable the network to bypass a large amount of low-frequency information and focus on learning more useful high-frequency information. The pixel and channel attention (PACA) network considers the interdependence between feature channels, while introducing fewer parameters and improving the network's representation ability. By combining the Inception structure with the attention mechanism, the network width is increased while also improving the network's applicability to different scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0021] Figure 1 2 is a schematic diagram of a super-resolution image reconstruction process according to an embodiment of the present invention;

[0022] Figure 2 2 is a schematic diagram of a super-resolution image reconstruction network structure according to an embodiment of the present invention;

[0023] Figure 3 Schematic diagram of the deep feature extraction model structure in an embodiment of the present invention;

[0024] Figure 4 Schematic diagram of the CA unit structure in an embodiment of the present invention;

[0025] Figure 5 Schematic diagram of the internal principle of the CA unit in an embodiment of the present invention;

[0026] Figure 6 Schematic diagram of the PA unit structure in an embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram of the Inception unit structure in an embodiment of the present invention;

[0028] FIG8( a ) is a low-resolution image to be reconstructed, and FIG8( b ) is a high-resolution image after reconstructing. DETAILED DESCRIPTION

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0031] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0032] Example 1

[0033] Vision is our primary means of acquiring information from the outside world. Images, as direct carriers of visual information, can accurately and intuitively reflect the various information contained within them. Image resolution refers to the number of pixels in an image and can be used to measure image quality. High-resolution images are rich in detail and carry a wealth of information, which not only provides a better visual experience but also facilitates our understanding and processing of the information contained in the image. Image super-resolution involves taking one or more low-resolution images as input and, through the use of specific algorithms and processing flows, combining image processing, artificial intelligence technology, computer vision, and other related knowledge, to reconstruct a corresponding high-resolution image based on the existing imaging system.

[0034] At present, image super-resolution methods can be mainly divided into the following three categories: interpolation-based super-resolution reconstruction, reconstruction-based super-resolution reconstruction, and learning-based super-resolution reconstruction.

[0035] (1) Interpolation-based SR reconstruction assumes that the newly added pixels in the image are only related to the pixel values ​​around the pixels. It mainly includes nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation.

[0036] (2) Reconstruction-based SR technology explores a large amount of image prior information to help complete the reconstruction of HR images. Classic reconstruction-based super-resolution algorithms include iterative back-projection method, convex set projection method, and maximum a posteriori probability method. This method has a high time complexity.

[0037] (3) Learning-based SR methods first divide the image data into blocks and construct sample libraries of low-resolution and high-resolution images respectively. Then, the corresponding mapping relationship between LR and HR is learned. Finally, based on the input low-resolution image, the learned mapping relationship is used to reconstruct the corresponding high-resolution image. Common learning-based methods include popular learning, sparse representation, and deep learning. Currently, the most widely studied and most effective method is the image super-resolution algorithm based on deep learning.

[0038] In recent years, with the advancement of deep learning, image super-resolution reconstruction technology has also progressed. In 2017, Lim et al. constructed a very wide network (EDSR) and a very deep network (MDSR) (approximately 165 layers) using simplified residual blocks. This method primarily modified ResNet. The authors removed some unnecessary modules from the residual structure, such as the batch normalization layer, and the results showed that this was indeed effective. Because batch normalization layers normalize features, it can get rid of the network's range variability by normalizing features, and it is best to remove them, thereby increasing the model size and improving the quality of the results. In addition, since the batch normalization layer consumes the same amount of memory as the preceding convolutional layer, removing the batch normalization layer also reduces the GPU memory usage of EDSR. The authors of EDSR believe that the simplest way to improve network model performance is to increase the number of parameters. This is achieved by stacking multiple layers or increasing the number of filters in a convolutional neural network.

[0039] However, simply stacking residual blocks to build deeper networks is unlikely to achieve better improvements. Whether deeper networks can further contribute to image SR and how to build very deep trainable networks remain to be explored.

[0040] like Figure 1 As shown, in order to solve the above problems, this embodiment provides an image super-resolution reconstruction method based on a residual channel attention network, including:

[0041] S1: Acquire the low-resolution image to be reconstructed;

[0042] S2: Obtaining a high-resolution reconstructed image based on the low-resolution image to be reconstructed and the image super-resolution reconstruction model; wherein the construction process of the image super-resolution reconstruction model includes:

[0043] S201: Obtain shallow features through the shallow feature channel, and use the first convolutional network to extract shallow features X0;

[0044] That is, the first convolutional network is represented as a convolutional layer:

[0045] X0= H SFE (I LR )(1)

[0046] Where HSFE ( ) represents the convolution function, and its kernel size is 3x3, so that the shallow feature X0 is obtained.

[0047] like Figure 2-Figure 3 As shown, S202: construct a deep feature extraction model based on the residual channel attention network, and extract deep features X1 according to the shallow features X0 and the deep feature extraction model;

[0048] The deep feature extraction model includes multiple pixel and channel attention (PACA) networks. Each of these networks includes a channel attention unit (CA) and a pixel attention unit (PA), along with an inception unit. These units are arranged in parallel, with a residual structure added to the outer layer. These units simultaneously process shallow features of the input and then fuse the processing results of each unit.

[0049] The process of image processing by the deep feature extraction model includes:

[0050] Based on the channel attention unit (CA), a one-dimensional (Cx1x1) attention feature vector is obtained by re-weighting the filter responses of all channels, selectively enhancing useful information and suppressing useless information;

[0051] Based on the pixel attention unit (PA), a 1×1 convolution layer and a sigmoid function are used to obtain the attention map, which is then multiplied with the input feature to generate a 3D (C × H × W) matrix as the attention feature.

[0052] Based on the Inception unit, 1x1 convolution is first used to reduce the number of channels for visual information aggregation. Then, features of different scales are extracted and pooled to obtain information at multiple scales. Finally, the features are superimposed and output. The information at multiple scales includes: Inception has multiple parallel branches. Feature superposition output is to superimpose the output features of these four branches.

[0053] like Figure 4-Figure 5 As shown, the specific process of obtaining the one-dimensional (Cx1x1) attention feature vector includes:

[0054] First, the channel-related global spatial information is converted into channel descriptors by using global average pooling, and X = [X1, ···, Xc, ···, XC] is taken as input, with C feature maps of size H × W. It is obtained by reducing X to the spatial size H × W.

[0055] Then the cth element of z is determined by the following formula (2):

[0056]

[0057] where Xc(i, j) is the value of the cth feature Xc at position (i, j), and HGP(·) represents the global pooling function. Such channel statistics can be viewed as a collection of local descriptors whose statistics help to express the entire image.

[0058] In order to fully capture the dependencies between channels from the aggregated information via global average pooling, a gating mechanism is introduced in CA.

[0059] The gating mechanism should satisfy two conditions: first, it must be able to learn nonlinear interactions between channels, and second, it must learn a non-mutually exclusive relationship, as it can emphasize multiple channel functions instead of activating them all at once.

[0060] In this embodiment, a simple gating mechanism with a sigmoid function is used, as shown in the following formula (3):

[0061]

[0062] in, and Represent the S-type gate and ReLU function respectively. W D is the set of weights of the Conv layer, which acts as a channel reduction with a reduction ratio r. After being activated by ReLU, the low-dimensional signal is then W U The channel amplification layer increases by a ratio r. Then, the final channel statistics s are obtained and used to rescale the input X c , that is, element-wise multiplication of the channel statistics and the input feature map of CA, After this series of operations, the network will focus on features with more useful information.

[0063] like Figure 6 As shown, the pixel attention unit (PA) includes: PA generates a 3D attention map instead of a 1D attention vector or a 2D map. This attention mechanism introduces fewer additional parameters but produces better SR results.

[0064] Channel attention aims to obtain a one-dimensional (C×1×1) attention feature vector, while spatial attention obtains a two-dimensional (1×H×W) attention map.

[0065] Where C is the number of channels, H and W are the height and width of the feature, respectively. Unlike these, pixel attention generates a 3D (C × H × W) matrix as the attention feature. In other words, pixel attention generates attention coefficients for all pixels in the feature map.

[0066] Pixel attention simply uses a 1×1 convolutional layer and a sigmoid function to obtain an attention map, which is then multiplied with the input features.

[0067] Denote the input and output feature maps as X k-1 and X k , the PA layer can be calculated as:

[0068]

[0069] in (·) is a 1 × 1 convolutional layer followed by a sigmoid function.

[0070] like Figure 7 As shown, the Inception unit (PA) includes:

[0071] Regarding inception, generally speaking, the safest way to improve network performance is to increase the width and depth of the network, which will also be accompanied by side effects.

[0072] First of all, deeper and wider networks often mean a huge number of parameters. When the amount of data is small, the trained network is prone to overfitting, and when the network is very deep, it is easy to cause gradient vanishing. These two side effects restrict the development of deep and wide convolutional neural networks. The Inception network solves these two problems very well.

[0073] The Inception structure first reduces the number of channels through 1x1 convolution to aggregate information, then extracts and pools features at different scales to obtain information at multiple scales, and finally superimposes and outputs the features. This increases the width of the network and its applicability to different scales.

[0074] S203: Expand the deep feature X1 to obtain the enlarged deep feature X2;

[0075] After an upsampling module, the shallow feature X1 is amplified and obtained

[0076] X2=HUS(X1)(5)

[0077] Where H US Represents the functionality of the upsampling module.

[0078] S204: reconstructing the amplified deep feature X2 through the second convolutional layer to obtain a high-resolution image with the same number of channels as the low-resolution image to be reconstructed;

[0079] X2 represents the features after amplification, which are then reconstructed through a convolutional layer.

[0080] I SR = H REC (X2) = H RPACA (I LR )(6)

[0081] Among them H REC ( ) represents the reconstruction layer, H RPACA () represents the function of the entire network.

[0082] S204: Use the loss function to optimize the proposed network structure. There are currently several loss functions, such as L1, L2, adversarial loss, and perceptual loss.

[0083] In order to demonstrate the effectiveness of the present invention, we use the L1 loss function.

[0084] Some existing image super-resolution reconstruction methods use the above-mentioned CA, PA and Inception modules, but they are all used separately in their respective networks. The present invention combines these three modules to enable them to play their respective advantages.

[0085] In this paper, all convolutional layers, except those after concat and upsampling, are resized to 3x3, with padding used to maintain a fixed size. The network is trained using 800 images from DIV2K and tested on four standard benchmark datasets: Set5, Set14, B100, and Urban100. SR results are evaluated on the Y channel of the transformed YCbCr space using PSNR and SSIM.

[0086] As shown in Figure 8 (a), it is the original low-resolution image to be reconstructed, and as shown in Figure 8 (b), it is the reconstructed high-resolution image. At the same time, the method proposed in this invention (RPACA) is compared with other methods, and the following quantitative analysis results are obtained.

[0087] Table 1 Quantitative comparison with other methods

[0088]

[0089] Example 2

[0090] This embodiment provides an image super-resolution reconstruction system based on a residual channel attention network, comprising: an image acquisition module, configured to: acquire a low-resolution image to be reconstructed;

[0091] The high-resolution image reconstruction module is configured to obtain a high-resolution reconstructed image based on a low-resolution image to be reconstructed and an image super-resolution reconstruction model; wherein the construction process of the image super-resolution reconstruction model includes: shallow feature extraction and deep feature extraction, wherein shallow features are obtained through a shallow feature channel, a deep feature extraction model is constructed based on a residual channel attention network, and deep features are extracted based on the shallow features and the deep feature extraction model. The deep feature extraction model includes multiple pixel and channel attention networks, and each pixel and channel attention network has a channel attention unit, a pixel attention unit, and an Inception unit arranged in parallel inside, and a residual structure is added in parallel on the outer layer.

[0092] Example 3

[0093] This embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the image super-resolution reconstruction method based on the residual channel attention network as described above.

[0094] Example 4

[0095] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the image super-resolution reconstruction method based on the residual channel attention network as described above are implemented.

[0096] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0097] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0098] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0100] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0101] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. Image super-resolution reconstruction method based on residual channel attention network, characterized by: The steps include: Acquire a low-resolution image to be reconstructed; A high-resolution reconstructed image is obtained based on a low-resolution image to be reconstructed and an image super-resolution reconstruction model; wherein the construction process of the image super-resolution reconstruction model includes: shallow feature extraction and deep feature extraction, wherein shallow features are obtained through a shallow feature channel, a deep feature extraction model is constructed based on a residual channel attention network, and deep features are extracted based on the shallow features and the deep feature extraction model. The deep feature extraction model includes multiple pixel and channel attention networks, each of which has a channel attention unit, a pixel attention unit, and an Inception unit arranged in parallel inside the pixel and channel attention network, and a residual structure is added in parallel on the outer layer; The process of image processing by the deep feature extraction model includes: Based on the channel attention unit, a one-dimensional attention feature vector is obtained by reweighting the filter responses of all channels; Based on the pixel attention unit, a 1×1 convolution layer and a sigmoid function are used to obtain the attention map, which is then multiplied with the input feature to generate a three-dimensional matrix as the attention feature; Based on the Inception unit, 1x1 convolution is used to reduce the number of channels for visual information aggregation, and feature extraction and pooling at different scales are performed to obtain information at multiple scales, and finally the features are superimposed and output.

2. The image super-resolution reconstruction method based on residual channel attention network according to claim 1, characterized in that The specific process of obtaining the one-dimensional attention feature vector includes: By using global average pooling, the channel-related global spatial information is converted into channel descriptors; A gating mechanism is introduced to fully capture the dependencies between channels from the aggregated information; The final channel statistics are obtained and used to rescale the position of the input features, and the channel statistics are multiplied with the input feature map of the channel attention unit.

3. The image super-resolution reconstruction method based on residual channel attention network according to claim 1, characterized in that The Inception unit includes multiple parallel convolutional layers, and Inception performs multiple convolution operations or pooling operations on the input image in parallel.

4. The image super-resolution reconstruction method based on residual channel attention network according to claim 1, characterized in that After obtaining the deep features, the deep features are expanded to obtain the amplified deep features; the amplified deep features are reconstructed through the convolution layer to obtain a high-resolution image with the same number of channels as the low-resolution image to be reconstructed.

5. The image super-resolution reconstruction method based on residual channel attention network according to claim 1, characterized in that During the training process of the image super-resolution reconstruction model, a loss function is used to constrain the difference between the high-definition reconstructed image and the low-resolution image to be reconstructed, and the parameters of the model are continuously adjusted until the model converges, thereby completing the model training.

6. Image super-resolution reconstruction system based on residual channel attention network, characterized by: Executing the image super-resolution reconstruction method based on the residual channel attention network according to any one of claims 1 to 5, comprising: The image acquisition module is configured to: acquire a low-resolution image to be reconstructed; The high-resolution image reconstruction module is configured to obtain a high-resolution reconstructed image based on a low-resolution image to be reconstructed and an image super-resolution reconstruction model; wherein the construction process of the image super-resolution reconstruction model includes: shallow feature extraction and deep feature extraction, wherein shallow features are obtained through a shallow feature channel, a deep feature extraction model is constructed based on a residual channel attention network, and deep features are extracted based on the shallow features and the deep feature extraction model. The deep feature extraction model includes multiple pixel and channel attention networks, and each pixel and channel attention network has a channel attention unit, a pixel attention unit, and an Inception unit arranged in parallel inside, and a residual structure is added in the outer layer in parallel.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the image super-resolution reconstruction method based on the residual channel attention network as described in any one of claims 1 to 5 are implemented.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the image super-resolution reconstruction method based on the residual channel attention network are implemented as described in any one of claims 1 to 5.

Images