Image super-resolution reconstruction method and system based on dilated residual dense convolution

By adopting hollow residual dense convolution technology in image super-resolution reconstruction, the problems of long network training time and underutilization of image features in the existing methods are solved, and the quality and efficiency of image reconstruction are improved.

CN114742701BActive Publication Date: 2024-11-29GUANGZHOU YINYUE SHIJIA INTELLIGENT TECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210209804.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-03
Publication Date
2024-11-29
Estimated Expiration
2042-03-03

AI Technical Summary

Technical Problem

The existing image super-resolution reconstruction method based on convolutional neural networks has problems such as long network training time, large amount of calculation and insufficient image features, resulting in poor image reconstruction quality and low efficiency.

Method used

Using the image super-resolution reconstruction method based on hollow residual dense convolution, the image reconstruction network is constructed, including feature extraction layer, nonlinear mapping layer and image reconstruction layer. The nonlinear mapping layer uses hollow residual dense module for feature mapping .

Benefits of technology

Under the same network parameters, the quality and efficiency of image super-resolution reconstruction are improved, the feature information of the image is fully utilized, and the difficulty of network training is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114742701B_ABST
    Figure CN114742701B_ABST
Patent Text Reader

Abstract

The present invention proposes an image super-resolution reconstruction method and system based on dilated residual dense convolution, including constructing an image reconstruction network including a feature extraction layer, a nonlinear mapping layer and an image reconstruction layer; the nonlinear mapping layer includes a dilated residual dense module composed of a plurality of dilated convolution layers and standard convolution layers densely connected in sequence; the preprocessed image to be reconstructed is input into the image reconstruction network, the image to be reconstructed is subjected to feature extraction and mapping with a high-resolution image, and the image reconstruction network outputs a high-resolution image. The residual structure composed of densely connected dilated convolution layers and standard convolution layers can extract more contextual information of the image and improve the reuse rate of feature information compared to the standard convolution layer, increase the contribution rate of the feature information extracted by the previous convolution layer to the final image reconstruction, make full use of the feature information of the image, reduce the difficulty of network training, and improve the quality and efficiency of image reconstruction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to an image super-resolution reconstruction method and system based on dilated residual dense convolution. Background Art

[0002] Image resolution is a set of performance parameters used to evaluate the richness of detail information contained in an image, including temporal resolution, spatial resolution, and color resolution, which reflects the ability of the imaging system to actually reflect the detail information of an object. Image super-resolution reconstruction technology refers to the process of reconstructing a high-resolution image from a given image to be reconstructed using technologies in the fields of digital image processing and computer vision, through specific algorithms and processing flows. Image super-resolution reconstruction technology aims to overcome or compensate for problems such as blurred images, low quality, and inconspicuous areas of interest caused by limitations of the image acquisition system or the acquisition environment itself.

[0003] There is an existing image super-resolution reconstruction method based on convolutional neural network, which uses the nearest neighbor interpolation algorithm to upsample the image, then uses a single-input and single-output feature enhancement layer to process the image, and then inputs the processed image into the convolutional neural network for feature mapping to complete the image super-resolution reconstruction. However, when using traditional convolutional neural networks to perform super-resolution reconstruction on images, there are problems such as long network training time, large amount of network model calculation, and insufficient utilization of image features, resulting in poor image reconstruction quality and low efficiency. Summary of the invention

[0004] In order to solve the defects of poor image super-resolution reconstruction quality and low efficiency in the prior art, the present invention provides an image super-resolution reconstruction method and system based on void residual dense convolution.

[0005] In a first aspect, the present invention proposes an image super-resolution reconstruction method based on dilated residual dense convolution, comprising the following steps:

[0006] S1: preprocessing the image to be reconstructed;

[0007] S2: constructing an image reconstruction network; the image reconstruction network comprises a feature extraction layer, a nonlinear mapping layer and an image reconstruction layer connected in sequence; the nonlinear mapping layer comprises a hole residual dense module, and the hole residual dense module comprises a plurality of hole convolution layers and standard convolution layers densely connected in sequence;

[0008] S3: inputting the preprocessed image to be reconstructed into the image reconstruction network, the feature extraction layer extracting feature information of the image to be reconstructed, and transmitting the feature information to the nonlinear mapping layer;

[0009] S4: the nonlinear mapping layer maps the feature information to the image reconstruction layer through a nonlinear function model, and the image reconstruction layer outputs a high-dimensional feature map;

[0010] S5: superimposing the image to be reconstructed with the high-dimensional feature map to obtain a high-resolution image.

[0011] As a preferred solution, the feature extraction layer includes a standard convolution layer Conv 1 The nonlinear mapping layer also includes a nonlinear mapping convolution module, and the hole residual dense module includes a hole convolution layer Conv that is densely connected in sequence. 2 , hole convolution layer Conv 3 , hole convolution layer Conv 4 And the standard convolution layer Conv 5 The nonlinear mapping convolution module includes n standard convolution layers Conv connected in sequence. m , wherein m=6, 7, ..., n+5; the image reconstruction layer includes a standard convolutional layer Conv';

[0012] The standard convolutional layer Conv 1 The output ends of the hole convolution layer Conv 2 , hole convolution layer Conv 3 , hole convolution layer Conv 4 , standard convolution layer Conv 5 The input terminal connection of

[0013] The hole convolution layer Conv 2 The output ends of the hole convolution layer Conv 3 , hole convolution layer Conv 4 , standard convolution layer Conv 5 The input terminal connection of

[0014] The hole convolution layer Conv 3 The output ends of the hole convolution layer Conv 4 And the standard convolution layer Conv 5 The input terminal connection of

[0015] The hole convolution layer Conv 4 The output of the standard convolutional layer Conv 5 The input terminal connection of

[0016] The standard convolutional layer Conv 5 The output end of is connected to the input end of the nonlinear mapping convolution module;

[0017] The output end of the nonlinear mapping convolution module is connected to the input end of the standard convolution layer Conv'.

[0018] As a preferred solution, the standard convolution layer Conv 1 , standard convolution layer Conv 5 And the standard convolutional layer Conv m The output ends of are all connected to an activation layer, and the activation layer is a ReLU function.

[0019] As a preferred solution, the step of inputting the preprocessed image to be reconstructed into the image reconstruction network to obtain a high-resolution image includes:

[0020] In the feature extraction layer, the image to be reconstructed Y is input into the standard convolution layer Conv 1 , standard convolutional layer Conv 1 Output feature map F 1 (Y), the specific operation expression is as follows:

[0021] F 1 (Y) = ReLU(W 1 *Y+B 1 )

[0022] Among them, W 1 represents the weight of the image reconstruction network, B 1 represents the bias value of the image reconstruction network, and ReLU(·) is a nonlinear activation function;

[0023] In the nonlinear mapping layer, the feature map F 1 (Y) Input hole convolution layer Conv 2 , the hole convolution layer Conv 2 Output feature map F 2 (Y); the feature map F 1 (Y) and feature map F 2 (Y) Input hole convolution layer Conv 3 , the hole convolution layer Conv 3 Output feature map F 3 (Y); the feature map F 1 (Y), feature map F 2 (Y) and feature map F 3 (Y) Input hole convolution layer Conv 4 , the hole convolution layer Conv 4 Output feature map F 4 (Y); the feature map F 1 (Y), feature map F 2 (Y), feature map F 3(Y) and feature map F 4 (Y) Input standard convolution layer Conv 5 , standard convolutional layer Conv 5 Output feature map F 5 (Y), the specific operation expression is as follows:

[0024] F i (Y) = ReLU(W i *{F i-1 (Y),F i-2 (Y),.....F 1 (Y)}+B i )

[0025] Where i represents the sequence number of the convolutional layer in the dilated residual dense module, and the value of i ranges from 2 to 5;

[0026] The feature map F 1 (Y) and feature map F 5 (Y) Perform superposition processing, input the superposition result into the nonlinear mapping convolution module, and the nonlinear mapping convolution module outputs the feature map F m (Y), the specific operation expression is as follows:

[0027] F m (Y) = ReLU(W m *F m-1 +B m )

[0028] In the image reconstruction layer, the feature map F m (Y) is input into the standard convolution layer Conv', and the standard convolution layer Conv' outputs a feature map F'(Y). The image to be reconstructed Y and the feature map F'(Y) are superimposed to obtain a high-resolution image I SR , the specific operation expression is as follows:

[0029] I SR =F'(Y)+Y.

[0030] As a preferred solution, the step of preprocessing the image to be reconstructed includes:

[0031] Perform bicubic interpolation and magnification processing on the image to be reconstructed so that the size of the image to be reconstructed is the same as the size of the target high-resolution image;

[0032] The image to be reconstructed after the amplification processing is converted into a color space to obtain a brightness channel image.

[0033] As a preferred solution, the image to be reconstructed is a low-resolution image.

[0034] In a second aspect, the present invention proposes an image super-resolution reconstruction system based on dilated residual dense convolution, which is applied to the image super-resolution reconstruction method based on dilated residual dense convolution described in any of the above schemes, including:

[0035] A preprocessing module, used for preprocessing the image to be reconstructed;

[0036] An image reconstruction network comprises a feature extraction layer, a nonlinear mapping layer and an image reconstruction layer connected in sequence; the nonlinear mapping layer comprises a hole residual dense module, and the hole residual dense module comprises a plurality of hole convolution layers and standard convolution layers densely connected in sequence;

[0037] The preprocessing module inputs the preprocessed image to be reconstructed into the image reconstruction network, the feature extraction layer extracts the feature information of the image to be reconstructed and transmits the feature information to the nonlinear mapping layer; the nonlinear mapping layer maps the feature information to the image reconstruction layer through a nonlinear function model, the image reconstruction layer outputs a high-dimensional feature map and superimposes the image to be reconstructed with the high-dimensional feature map to obtain a high-resolution image.

[0038] As a preferred solution, the preprocessing module includes a bicubic interpolation and magnification unit, which is used to perform bicubic interpolation and magnification processing on the input image to be reconstructed so that the size of the image to be reconstructed is the same as the size of the target high-resolution image; and perform color space conversion on the image to be reconstructed after the magnification processing to obtain a brightness channel image.

[0039] As a preferred solution, the feature extraction layer includes a standard convolution layer Conv 1 The nonlinear mapping layer also includes a nonlinear mapping convolution module, and the hole residual dense module includes a hole convolution layer Conv that is densely connected in sequence. 2 , hole convolution layer Conv 3 , hole convolution layer Conv 4 And the standard convolution layer Conv 5 The nonlinear mapping convolution module includes n standard convolution layers Conv connected in sequence. m , wherein m=6, 7, ..., n+5; the image reconstruction layer includes a standard convolutional layer Conv';

[0040] The standard convolutional layer Conv 1 The output ends of the hole convolution layer Conv 2 , hole convolution layer Conv 3 , hole convolution layer Conv 4 , standard convolution layer Conv 5 The input terminal connection of

[0041] The hole convolution layer Conv 2 The output ends of the hole convolution layer Conv 3 , hole convolution layer Conv 4 , standard convolution layer Conv 5 The input terminal connection of

[0042] The hole convolution layer Conv 3 The output ends of the hole convolution layer Conv 4 And the standard convolution layer Conv 5 The input terminal connection of

[0043] The hole convolution layer Conv 4 The output of the standard convolutional layer Conv 5 The input terminal connection of

[0044] The standard convolutional layer Conv 5 The output end of is connected to the input end of the nonlinear mapping convolution module;

[0045] The output end of the nonlinear mapping convolution module is connected to the standard convolution layer Conv m The input terminal connection of

[0046] As a preferred solution, the standard convolution layer Conv 1 , standard convolution layer Conv 5 And the standard convolutional layer Conv m The output ends of are all connected to an activation layer, and the activation layer is a ReLU function.

[0047] In a third aspect, the present invention proposes a computer system comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program in the memory, the steps described in any scheme of the image super-resolution reconstruction method based on void residual dense convolution are implemented.

[0048] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: a dilated residual dense module including a plurality of sequentially densely connected dilated convolutional layers and standard convolutional layers is used to perform nonlinear mapping on the feature data of the image to be reconstructed. Under the condition of the same network parameters, more contextual information of the image can be extracted and the reuse rate of the feature information can be improved compared with only the standard convolutional layer. The contribution rate of the feature information extracted by the previous convolutional layer to the final image reconstruction is increased, the feature information of the image is fully utilized, the difficulty of network training is reduced, and the quality and efficiency of image reconstruction are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a schematic diagram of the hollow residual dense module in Example 1.

[0050] Figure 2 Flowchart of the image super-resolution reconstruction method based on void residual dense convolution.

[0051] Figure 3 This is a schematic diagram of the image reconstruction network in Example 2.

[0052] Figure 4 Schematic diagram of the setting of convolution kernel parameters in the image reconstruction network in Example 2.

[0053] Figure 5 This is the architecture diagram of the image super-resolution reconstruction system based on void residual dense convolution. DETAILED DESCRIPTION

[0054] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0055] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0056] Example 1

[0057] See also Figure 1-Figure 2 This embodiment proposes an image super-resolution reconstruction method based on dilated residual dense convolution, including:

[0058] S1: Preprocess the image to be reconstructed.

[0059] S2: construct an image reconstruction network; the image reconstruction network includes a feature extraction layer, a nonlinear mapping layer and an image reconstruction layer connected in sequence; the nonlinear mapping layer includes a hole residual dense module, and the hole residual dense module includes a plurality of hole convolution layers and standard convolution layers densely connected in sequence.

[0060] S3: The preprocessed image to be reconstructed is input into the image reconstruction network, the feature extraction layer extracts feature information of the image to be reconstructed, and transmits the feature information to the nonlinear mapping layer.

[0061] S4: The nonlinear mapping layer maps the feature information to an image reconstruction layer through a nonlinear function model, and the image reconstruction layer outputs a high-dimensional feature map.

[0062] S5: superimposing the image to be reconstructed with the high-dimensional feature map to obtain a high-resolution image.

[0063] In this embodiment, a dilated residual dense block (DRDBlock) is used to perform nonlinear mapping on the feature data of the image to be reconstructed. The dilated residual dense block is composed of a dilated convolutional layer Conv2 , hole convolution layer Conv 3 , hole convolution layer Conv 4 And the standard convolution layer Conv 5 The residual structure composed of dense connections in sequence, such as Figure 1 As shown, Figure 1 The schematic diagram of the hole residual dense module in this embodiment, the hole convolution layer Conv 2 The output ends of the hole convolution layer Conv 3 , hole convolution layer Conv 4 , standard convolution layer Conv 5 The input end of the hole convolution layer Conv 3 The output ends of the hole convolution layer Conv 4 And the standard convolution layer Conv 5 The input end of the hole convolution layer Conv 4 The output of the standard convolutional layer Conv 5 The input terminal connection of

[0064] In the specific implementation process, the feature extraction layer extracts the feature information of the image to be reconstructed, and inputs the feature information into the hole residual dense module for nonlinear mapping: First, the feature map F 1 (Y) Input hole convolution layer Conv 2 , the hole convolution layer Conv 2 Output feature map F 2 (Y); the feature map F 1 (Y) and feature map F 2 (Y) Input hole convolution layer Conv 3 , the hole convolution layer Conv 3 Output feature map F 3 (Y); the feature map F 1 (Y), feature map F 2 (Y) and feature map F 3 (Y) Input hole convolution layer Conv 4 , the hole convolution layer Conv 4 Output feature map F 4 (Y); the feature map F 1 (Y), feature map F 2 (Y), feature map F 3 (Y) and feature map F 4 (Y) Input standard convolution layer Conv 5 , standard convolutional layer Conv 5 Output feature map F 5(Y); the feature map F 1 (Y) is superimposed with the feature map F(Y), and the superposition result is used as the output data of the void residual dense module, which is input into a nonlinear mapping convolution module. The nonlinear mapping convolution module outputs the final mapping data and transmits it to the image reconstruction layer. The image reconstruction layer outputs a high-dimensional feature map, and the image to be reconstructed is superimposed with the high-dimensional feature map to obtain a high-resolution image.

[0065] Under the same network parameters, compared with the standard convolution layer, the receptive field of the hole convolution layer is larger and it can extract more contextual information of the image; using a densely connected residual structure between the hole convolution layer and the standard convolution layer, and using the feature information extracted by the previous convolution layer in the same residual structure as the input of the next convolution layer, can improve the reuse rate of feature information, increase the contribution rate of the feature information extracted by the previous convolution layer to the final image reconstruction, make full use of the feature information of the image, and reduce the difficulty of network training. Therefore, using a residual structure composed of densely connected hole convolution layers and standard convolution layers to perform nonlinear mapping on the feature data of the image to be reconstructed can improve the quality and efficiency of image reconstruction.

[0066] Example 2

[0067] See also Figure 3-Figure 4 This embodiment makes improvements on the image super-resolution reconstruction method based on hole residual dense convolution proposed in Embodiment 1.

[0068] In this embodiment, preprocessing the image to be reconstructed includes the following steps:

[0069] The image to be reconstructed is enlarged by bicubic interpolation so that the size of the image to be reconstructed is the same as the size of the target high-resolution image. The enlarged image to be reconstructed is converted into a color space to obtain a brightness channel image.

[0070] like Figure 3 and Figure 4 As shown, Figure 3 is a schematic diagram of the image reconstruction network in this embodiment, Figure 4 Schematic diagram of the setting of convolution kernel parameters in the image reconstruction network in this embodiment. The feature extraction layer includes a standard convolution layer Conv 1 The nonlinear mapping layer also includes a nonlinear mapping convolution module, and the hole residual dense module includes a hole convolution layer Conv that is densely connected in sequence. 2 , hole convolution layer Conv 3 , hole convolution layer Conv 4 And the standard convolution layer Conv 5 The nonlinear mapping convolution module includes n standard convolution layers Conv connected in sequence.m , where m=6, 7, ..., n+5; the image reconstruction layer includes a standard convolution layer Conv'. The standard convolution layer Conv 1 The output ends of the hole convolution layer Conv 2 , hole convolution layer Conv 3 , hole convolution layer Conv 4 , standard convolution layer Conv 5 The hole convolution layer Conv 2 The output ends of the hole convolution layer Conv 3 , hole convolution layer Conv 4 , standard convolution layer Conv 5 The hole convolution layer Conv 3 The output ends of the hole convolution layer Conv 4 And the standard convolution layer Conv 5 The hole convolution layer Conv 4 The output of the standard convolutional layer Conv 5 The standard convolution layer Conv 5 The output end of is connected to the input end of the nonlinear mapping convolution module. The output end of the nonlinear mapping convolution module is connected to the input end of the standard convolution layer Conv'.

[0071] In the feature extraction layer, the image to be reconstructed Y is input into the standard convolution layer Conv 1 , standard convolutional layer Conv 1 Output feature map F 1 (Y); Standard convolution layer Conv 1 Use 64 3×3 convolution kernels to extract the feature information of the brightness channel image, making the low-dimensional feature information richer. The specific operation expression is as follows:

[0072] F 1 (Y) = ReLU(W 1 *Y+B 1 )

[0073] Among them, W 1 represents the weight of the image reconstruction network, B 1 Represents the bias value of the image reconstruction network. W 1 The size of is set to Conv×n×f×f, where Conv represents the number of input channels, n represents the number of convolution kernels, i.e., the number of output channels, f×f represents the size of the convolution kernel, and ReLU(·) is the nonlinear activation function.

[0074] In the nonlinear mapping layer, the hole convolution layer Conv2 , hole convolution layer Conv 3 And the hole convolution layer Conv 4 The void ratio of is 2, and each layer uses 32 3×3 convolution kernels to extract deep features of the image. 1 (Y) Input hole convolution layer Conv 2 , the hole convolution layer Conv 2 Output feature map F 2 (Y); the feature map F 1 (Y) and feature map F 2 (Y) Input hole convolution layer Conv 3 , the hole convolution layer Conv 3 Output feature map F 3 (Y); the feature map F 1 (Y), feature map F 2 (Y) and feature map F 3 (Y) Input hole convolution layer Conv 4 , the hole convolution layer Conv 4 Output feature map F 4 (Y); the feature map F 1 (Y), feature map F 2 (Y), feature map F 3 (Y) and feature map F 4 (Y) Input standard convolution layer Conv 5 , standard convolutional layer Conv 5 Output feature map F 5 (Y). Standard convolutional layer Conv 2 Use 64 3×3 convolution kernels. The specific operation expression is as follows:

[0075] F i (Y) = ReLU(W i *{F i-1 (Y),F i-2 (Y),.....F 1 (Y)}+B i )

[0076] Here, i represents the sequence number of the convolutional layer in the DRD Block, and the value range of i is 2 to 5. In the DRD Block, the output of the i-th layer is related to the previous i-1, i-2, .....1 layers, so the utilization rate of feature information between convolutional layers can be improved.

[0077] In this embodiment, the nonlinear mapping convolution module includes 18 standard convolution layers, namely Conv6, Conv7, ..., Conv23, and each of the 18 standard convolution layers uses 64 3×3 convolution kernels. 1 (Y) and feature map F 5 (Y) Perform superposition processing, input the superposition result into the nonlinear mapping convolution module, and the nonlinear mapping convolution module outputs the feature map F 23 (Y), the specific operation expression is as follows:

[0078] F m (Y) = ReLU(W m *F m-1 +B m )

[0079] Where m represents the number of the convolution layer in the nonlinear mapping convolution module, and the value of m ranges from 6 to 23. 5 (Y) and F 1 The superposition result of (Y) is used as Conv 6 The input is convolved and the output feature map F 6 (Y) Conv 6 To Conv 23 All of them use 64 3×3 convolution kernels for feature mapping. Therefore, the final output feature map F of the nonlinear mapping layer is obtained by analogy. 23 (Y).

[0080] In the image reconstruction layer, the feature map F 23 (Y) Input the standard convolution layer Conv', which uses a 3×3 convolution kernel to perform high-dimensional feature mapping on the high-dimensional feature map F 23 (Y) are arranged and combined to obtain a feature map F'(Y) containing high-frequency information of the reconstructed image, and the image to be reconstructed Y and the feature map F'(Y) are superimposed to obtain a high-resolution image I SR The specific operation expression is as follows:

[0081] I SR =F'(Y)+Y.

[0082] The present invention constructs an image reconstruction network with 24 convolutional layers, which improves the quality of reconstructed images. Compared with traditional image reconstruction methods, the quality of reconstructed images has been significantly improved, especially in terms of detail texture and high-frequency information. The present invention uses a hole convolution layer to replace part of the standard convolution layer, which expands the receptive field under the same convolution kernel parameters; at the same time, the residual structure is used to strengthen the information transmission between the bottom network layer and the high-level network layer, which reduces the difficulty of network training and alleviates the problems of gradient disappearance and gradient explosion; in addition, a dense connection structure is used between the hole convolution layer and the standard convolution layer, which improves the utilization rate of feature information between convolution layers, so that the feature information extracted by the previous convolution layer can contribute to the final image reconstruction.

[0083] Example 3

[0084] See also Figure 5 , this embodiment proposes an image super-resolution reconstruction system based on void residual dense convolution, including a preprocessing module 1 and an image reconstruction network 2.

[0085] The preprocessing module 1 includes a bicubic interpolation and magnification unit. In the specific implementation process, the bicubic interpolation and magnification unit performs bicubic interpolation and magnification processing on the input image to be reconstructed so that the size of the image to be reconstructed is the same as the size of the target high-resolution image; the image to be reconstructed after the magnification processing is converted into a color space to obtain a brightness channel image. The brightness channel image is input into the image reconstruction network 2. The image reconstruction module includes a feature extraction layer 21, a nonlinear mapping layer 22 and an image reconstruction layer 23 connected in sequence. The nonlinear mapping layer 22 includes a hole residual dense module, and the hole residual dense module includes a plurality of hole convolution layers and standard convolution layers that are densely connected in sequence. The plurality of hole convolution layers and standard convolution layers that are densely connected in sequence form a residual structure.

[0086] In this embodiment, the feature extraction layer 21 includes a standard convolutional layer Conv 1 The nonlinear mapping layer 22 also includes a nonlinear mapping convolution module, and the hole residual dense module includes a hole convolution layer Conv 2 , hole convolution layer Conv 3 , hole convolution layer Conv 4 And the standard convolution layer Conv 5 The nonlinear mapping convolution module includes n standard convolution layers Conv connected in sequence. m , where m=6, 7, ..., n+5; the image reconstruction layer 23 includes a standard convolution layer Conv'. The standard convolution layer Conv 1 The output ends of the hole convolution layer Conv 2 , hole convolution layer Conv3 , hole convolution layer Conv 4 , standard convolution layer Conv 5 The hole convolution layer Conv 2 The output ends of the hole convolution layer Conv 3 , hole convolution layer Conv 4 , standard convolution layer Conv 5 The hole convolution layer Conv 3 The output ends of the hole convolution layer Conv 4 And the standard convolution layer Conv 5 The hole convolution layer Conv 4 The output of the standard convolutional layer Conv 5 The standard convolution layer Conv 5 The output end of is connected to the input end of the nonlinear mapping convolution module. The output end of the nonlinear mapping convolution module is connected to the input end of the standard convolution layer Conv'.

[0087] In the specific implementation process, the brightness channel image Y is input into the standard convolution layer Conv 1 , standard convolutional layer Conv 1 Output feature map F 1 (Y).

[0088] The feature map F 1 (Y) Input hole convolution layer Conv 2 , the hole convolution layer Conv 2 Output feature map F 2 (Y). The feature map F 1 (Y) and feature map F 2 (Y) Input hole convolution layer Conv 3 , the hole convolution layer Conv 3 Output feature map F 3 (Y). The feature map F 1 (Y), feature map F 2 (Y) and feature map F 3 (Y) Input hole convolution layer Conv 4 , the hole convolution layer Conv 4 Output feature map F 4 (Y). The feature map F 1 (Y), feature map F 2 (Y), feature map F 3 (Y) and feature map F 4 (Y) Input standard convolution layer Conv5 , standard convolutional layer Conv 5 Output feature map F 5 (Y). The feature map F 1 (Y) and feature map F 5 (Y) Perform superposition processing, input the superposition result into the nonlinear mapping convolution module, and the nonlinear mapping convolution module outputs the feature map F m (Y). The feature map F m (Y) inputs the standard convolution layer Conv', the standard convolution layer Conv' outputs a feature map F'(Y), the image to be reconstructed and the feature map F'(Y) are superimposed to obtain a high-resolution image I SR .

[0089] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0090] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An image super-resolution reconstruction method based on dilated residual dense convolution, characterized in that: The following steps are involved: S1: preprocessing the image to be reconstructed; S2: constructing an image reconstruction network; the image reconstruction network comprises a feature extraction layer, a nonlinear mapping layer and an image reconstruction layer connected in sequence; the nonlinear mapping layer comprises a hole residual dense module, and the hole residual dense module comprises a plurality of hole convolution layers and standard convolution layers densely connected in sequence; S3: inputting the preprocessed image to be reconstructed into the image reconstruction network, the feature extraction layer extracting feature information of the image to be reconstructed, and transmitting the feature information to the nonlinear mapping layer; S4: the nonlinear mapping layer maps the feature information to the image reconstruction layer through a nonlinear function model, and the image reconstruction layer outputs a high-dimensional feature map; S5: superimposing the image to be reconstructed with the high-dimensional feature map to obtain a high-resolution image; Inputting the preprocessed image to be reconstructed into the image reconstruction network to obtain a high-resolution image, including: In the feature extraction layer, the image Y to be reconstructed is input into the standard convolution layer Conv1, and the standard convolution layer Conv1 outputs the feature map F1(Y). The specific operation expression is as follows: F1(Y)=ReLU(W1*Y+B1) Where W1 represents the weight of the image reconstruction network, B1 represents the bias value of the image reconstruction network, and ReLU (·) is a nonlinear activation function; In the nonlinear mapping layer, the feature map F1(Y) is input into the hole convolution layer Conv2, and the hole convolution layer Conv2 outputs the feature map F2(Y); the feature map F1(Y) and the feature map F2(Y) are input into the hole convolution layer Conv3, and the hole convolution layer Conv3 outputs the feature map F3(Y); the feature map F1(Y), the feature map F2(Y) and the feature map F3(Y) are input into the hole convolution layer Conv4, and the hole convolution layer Conv4 outputs the feature map F4(Y); the feature map F1(Y), the feature map F2(Y), the feature map F3(Y) and the feature map F4(Y) are input into the standard convolution layer Conv5, and the standard convolution layer Conv5 outputs the feature map F5(Y). The specific operation expressions are as follows: F i (Y)=ReLU(W i *{F i-1 (Y),F i-2 (Y),.....F1(Y)}+B i ) Where i represents the sequence number of the convolutional layer in the dilated residual dense module, and the value of i ranges from 2 to 5; The feature map F1(Y) is superimposed with the feature map F5(Y), and the superimposed result is input into the nonlinear mapping convolution module, which outputs the feature map F m (Y), the specific operation expression is as follows: F m (Y)=ReLU(W m *F m-1 +B m ) In the image reconstruction layer, the feature map F m (Y) inputs the standard convolution layer Conv', the standard convolution layer Conv' outputs the feature map F'(Y), the image to be reconstructed Y and the feature map F'(Y) are superimposed to obtain a high-resolution image I SR , the specific operation expression is as follows: I SR =F'(Y)+Y 。 2. The image super-resolution reconstruction method based on dilated residual dense convolution according to claim 1, characterized in that: The feature extraction layer includes a standard convolution layer Conv1, the nonlinear mapping layer also includes a nonlinear mapping convolution module, the hole residual dense module includes a hole convolution layer Conv2, a hole convolution layer Conv3, a hole convolution layer Conv4 and a standard convolution layer Conv5 which are densely connected in sequence, and the nonlinear mapping convolution module includes n standard convolution layers Conv connected in sequence. m , where m=6,7,...,n+5; the image reconstruction layer includes a standard convolutional layer Conv'; The output end of the standard convolution layer Conv1 is connected to the input ends of the dilated convolution layer Conv2, the dilated convolution layer Conv3, the dilated convolution layer Conv4, and the standard convolution layer Conv5 respectively; The output end of the dilated convolution layer Conv2 is connected to the input ends of the dilated convolution layer Conv3, the dilated convolution layer Conv4, and the standard convolution layer Conv5 respectively; The output end of the hole convolution layer Conv3 is connected to the input end of the hole convolution layer Conv4 and the standard convolution layer Conv5 respectively; The output end of the hole convolution layer Conv4 is connected to the input end of the standard convolution layer Conv5; The output end of the standard convolution layer Conv5 is connected to the input end of the nonlinear mapping convolution module; The output end of the nonlinear mapping convolution module is connected to the input end of the standard convolution layer Conv'.

3. The image super-resolution reconstruction method based on dilated residual dense convolution according to claim 2, characterized in that: The standard convolution layer Conv1, the standard convolution layer Conv5 and the standard convolution layer Conv m The output ends of are all connected to an activation layer, and the activation layer is a ReLU function.

4. The image super-resolution reconstruction method based on dilated residual dense convolution according to any one of claims 1 to 3, characterized in that: The steps for preprocessing the image to be reconstructed include: Perform bicubic interpolation and magnification processing on the image to be reconstructed so that the size of the image to be reconstructed is the same as the size of the target high-resolution image; The image to be reconstructed after the amplification processing is converted into a color space to obtain a brightness channel image.

5. An image super-resolution reconstruction system based on dilated residual dense convolution, applied to the image super-resolution reconstruction method based on dilated residual dense convolution as claimed in any one of claims 1 to 4, characterized in that: include: A preprocessing module, used for preprocessing the image to be reconstructed; An image reconstruction network comprises a feature extraction layer, a nonlinear mapping layer and an image reconstruction layer connected in sequence; the nonlinear mapping layer comprises a hole residual dense module, and the hole residual dense module comprises a plurality of hole convolution layers and standard convolution layers densely connected in sequence; The preprocessing module inputs the preprocessed image to be reconstructed into the image reconstruction network, and the feature extraction layer extracts feature information of the image to be reconstructed and transmits the feature information to the nonlinear mapping layer; The nonlinear mapping layer maps the feature information to the image reconstruction layer through a nonlinear function model. The image reconstruction layer outputs a high-dimensional feature map and superimposes the image to be reconstructed with the high-dimensional feature map to obtain a high-resolution image.

6. The image super-resolution reconstruction system based on dilated residual dense convolution according to claim 5, characterized in that: The preprocessing module includes a bicubic interpolation and magnification unit, which is used to perform bicubic interpolation and magnification processing on the input image to be reconstructed so that the size of the image to be reconstructed is the same as the size of the target high-resolution image; and perform color space conversion on the image to be reconstructed after the magnification processing to obtain a brightness channel image.

7. The image super-resolution reconstruction system based on dilated residual dense convolution according to claim 5, characterized in that: The feature extraction layer includes a standard convolution layer Conv1, the nonlinear mapping layer also includes a nonlinear mapping convolution module, the hole residual dense module includes a hole convolution layer Conv2, a hole convolution layer Conv3, a hole convolution layer Conv4 and a standard convolution layer Conv5 which are densely connected in sequence, and the nonlinear mapping convolution module includes n standard convolution layers Conv connected in sequence. m , where m=6,7,...,n+5; the image reconstruction layer includes a standard convolutional layer Conv'; The output end of the standard convolution layer Conv1 is connected to the input ends of the dilated convolution layer Conv2, the dilated convolution layer Conv3, the dilated convolution layer Conv4, and the standard convolution layer Conv5 respectively; The output end of the dilated convolution layer Conv2 is connected to the input ends of the dilated convolution layer Conv3, the dilated convolution layer Conv4, and the standard convolution layer Conv5 respectively; The output end of the hole convolution layer Conv3 is connected to the input end of the hole convolution layer Conv4 and the standard convolution layer Conv5 respectively; The output end of the hole convolution layer Conv4 is connected to the input end of the standard convolution layer Conv5; The output end of the standard convolution layer Conv5 is connected to the input end of the nonlinear mapping convolution module; The output end of the nonlinear mapping convolution module is connected to the standard convolution layer Conv m The input terminal connection of 8. The image super-resolution reconstruction system based on dilated residual dense convolution according to claim 7, characterized in that: The standard convolution layer Conv1, the standard convolution layer Conv5 and the standard convolution layer Conv m The output ends of are all connected to an activation layer, and the activation layer is a ReLU function.

9. A computer system, characterized in that: It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program in the memory, the steps of the image super-resolution reconstruction method based on hole residual dense convolution as claimed in any one of claims 1 to 4 are implemented.

Citation Information

Patent Citations

  • Residual convolutional neural network-based image super resolution reconstruction method

    CN108921786A

  • Pixel-level classification method, device and equipment and storage medium

    CN111382759A