An image super-resolution reconstruction method based on instance space feature modulation and feedback mechanism

Abstract

The present invention discloses an image super-resolution reconstruction method based on instance space feature modulation and feedback mechanism, which is used to solve the problem of reconstructing low-resolution images into high-resolution images. First, the instance space features of the low-resolution image are extracted by an instance space feature modulator, and the super-resolution reconstruction features of the low-resolution image are characterized by instance space features based on the extracted features. Then, the modulated super-resolution reconstruction features are input into a feedback reconstruction subnet including a projection module for cyclic reconstruction. After multiple reconstruction iterative optimizations, the final super-resolution reconstruction results are obtained. Introducing instance space features into super-resolution reconstruction features and using a recurrent neural network for multiple reconstruction iterative optimizations will not only better reconstruct low-resolution images, but also have significant effects on the texture information of image instances, greatly improving the effect of image reconstruction.

Description

Technical Field

[0001] The invention belongs to the field of image processing and computer vision, and relates to a method for image super-resolution reconstruction, in particular to an image super-resolution reconstruction method based on instance space feature modulation and feedback mechanism. Background Art

[0002] Image super-resolution technology refers to the reconstruction of corresponding high-resolution images from observed low-resolution images, and has important application value in monitoring equipment, satellite images, medical imaging and other fields.

[0003] After 2014, deep learning was introduced into the field of image super-resolution and developed rapidly. This method requires the use of interpolation to assist in enlarging the original low-resolution image before the neural network, and then enters the deep neural network to let it learn the super-resolution reconstruction method. Later in 2016, the previous method was improved. There was no need to enlarge the image outside the deep neural network, but directly input the original low-resolution image. By changing the mapping layer and feature dimension, the training time of the model was greatly reduced, and it was made universal, greatly improving the reconstruction effect. Later, the adversarial neural network method was introduced into the field of super-resolution reconstruction. The previous methods were to optimize and improve the accuracy of the deep neural network by comparing several pairs of low-resolution images. The adversarial idea of ​​the adversarial neural network improves the efficiency of this optimization and reduces the dependence on the data set, greatly improving the super-resolution reconstruction effect.

[0004] Although research results have shown that adding semantic information to the super-resolution reconstruction process can achieve high-quality super-resolution reconstruction, it is still quite challenging to reconstruct instance-level semantic features in the image and texture information in the original image with lower resolution. In addition, in many tasks, low-resolution images will lose image information due to various reasons, resulting in the loss of image information in the reconstruction process.

[0005] In order to solve the above problems, the present invention proposes an image super-resolution reconstruction method based on instance space feature modulation and feedback mechanism. Summary of the invention

[0006] The present invention is used to introduce instance-level semantic features of images in the process of super-resolution reconstruction, so as to improve the reconstruction effect of the original image with lower resolution and retain more texture information. The present invention provides an image super-resolution reconstruction method based on instance space feature modulation and feedback mechanism. The present invention realizes the reconstruction of a clear high-resolution image in a relatively low-resolution image. First, the instance space features of the low-resolution image are extracted by an instance space feature modulator, and the super-resolution reconstruction features of the low-resolution image are characterized by instance space features based on the extracted features. Then, the modulated super-resolution reconstruction features are input into the feedback reconstruction subnet including the projection module for cyclic reconstruction. After multiple reconstruction iterative optimizations, the final super-resolution reconstruction result is obtained. Introducing instance space features in the super-resolution reconstruction features and using a recurrent neural network for multiple reconstruction iterative optimizations will not only better reconstruct the low-resolution image, but also have a significant effect on the texture information of the image instance, greatly improving the image reconstruction effect. Specifically comprising the following steps:

[0007] 1) Input a low-resolution image and extract features as the initial super-resolution reconstruction features;

[0008] 2) Design an instance space feature modulator, and use the instance information in the instance segmentation map corresponding to the input low-resolution image as the prior knowledge for super-resolution reconstruction. The instance space feature modulator extracts the instance space features of the low-resolution image, modulates the initial super-resolution reconstruction features by the dot product, and obtains the modulated image super-resolution reconstruction features. By modulating the initial super-resolution reconstruction features by the instance space feature, the model not only "knows" how to reconstruct, but also "knows" where to reconstruct.

[0009] 3) Based on the loop feedback mechanism, a feedback reconstruction subnet is designed and a projection module is introduced. The modulated image super-resolution reconstruction features are input into the feedback reconstruction subnet. A reconstruction result is obtained through up / down sampling operations at different scales in the projection module, and the reconstruction result is iteratively optimized multiple times based on the loop feedback mechanism to obtain the final reconstruction result.

[0010] Furthermore, in step 1), a low-resolution image is input and image features are extracted, specifically:

[0011] 1) First, the input of the model is a low-resolution image and its corresponding instance segmentation map. The low-resolution image is passed through a 1×1 convolution layer to obtain a 256×256×3 feature map as the initial super-resolution reconstruction feature;

[0012] Further, the instance space feature modulator in step 2) obtains the instance space feature, specifically:

[0013] 2.1) For the image super-resolution reconstruction task, the spatial information of the image is very important. The instance space feature modulator in the model of the present invention extracts the instance space features of the instance segmentation map, and modulates the initial super-resolution reconstruction features based on the extracted instance space features to enhance the influence of the instance space features in the super-resolution reconstruction process, thereby overcoming the problem that different instances with the same semantics cannot be effectively distinguished by relying solely on semantic category information.

[0014] The specific instance spatial feature modulator is composed of a convolutional layer and a two-dimensional Gaussian distribution layer.

[0015] First, the instance space feature modulator performs a 3×3 convolution on the input instance segmentation map to extract the instance space features of the image; then, a two-dimensional Gaussian distribution is used to encode the extracted instance space features to generate an instance space heat map HP:

[0016]

[0017] in, Represents the instance segmentation map corresponding to the low-resolution image, Conv 3×3 represents a 3×3 convolution operation, and Gau represents a two-dimensional Gaussian distribution.

[0018] 2.2) Based on the instance space heat map HP containing instance space feature information, the initial super-resolution reconstruction feature is modulated to enhance the instance space feature information in the super-resolution reconstruction feature. After downsampling, the modulated super-resolution reconstruction feature of size 64×64×3 is obtained as the input F of the feedback reconstruction subnet. in :

[0019] F in =f(x·HP) (2)

[0020] Among them, HP is the instance space heat map containing instance space feature information, x is the initial super-resolution reconstruction feature obtained by feature extraction of the low-resolution image, · represents the multiplication of corresponding elements of the matrix, and f represents two 3×3 convolution operations.

[0021] F in It is the super-resolution reconstruction feature after instance space feature modulation, which not only contains the low-resolution image features, but also has the features of the instance segmentation map and instance space feature information through the instance space feature modulator.

[0022] Furthermore, in step 3), a feedback reconstruction subnet is designed to perform multiple iterative reconstruction optimizations on the super-resolution reconstruction features after instance space feature modulation.

[0023] The processing of the tth iteration of the feedback reconstruction subnet is as follows:

[0024] 3.1) The super-resolution reconstruction feature F modulated by the instance space feature in As the input of the feedback reconstruction subnet, after being processed by the feedback reconstruction subnet, the output is obtained

[0025]

[0026] in, is the output of the t-1th iteration feedback reconstruction subnetwork, F in is the super-resolution reconstruction feature modulated by the instance space feature, P t It is the projection module in the feedback reconstruction subnet.

[0027] Among them, the super-resolution reconstruction feature F in And the output of the t-1th iteration feedback reconstruction subnetwork As the input of the projection module, it is processed as follows;

[0028] 3.1.1) First, F in and Perform concat operation and then obtain H after 1×1 convolution t :

[0029]

[0030] 3.1.2) Then, H t Obtain high-resolution features through deconvolution The low-resolution features are obtained through 3×3 convolution and combined with H t Subtract to obtain residual features

[0031]

[0032]

[0033] Residual features It contains some feature information that is not shared by high-level features and low-level features, so that the feedback reconstruction subnet can better refine the low-level feature information.

[0034] 3.1.3) Then the residual feature After 1×1 convolution, deconvolution is performed and combined with high-resolution features Fusion, get the fused features

[0035]

[0036] 3.1.4) The projection module repeats the feature residual-fusion operations of 3.1.1, 3.1.2 and 3.1.3 three times, and finally obtains the refined high-level features That is, as the output of the projection module for the tth time

[0037] By performing residual-fusion operation on the features, the output Compensation information at different scales can be obtained, thereby containing more accurate instance space location information.

[0038] 3.2) After two upsampling operations, the super-resolution reconstruction result SR of the tth iteration is obtained. t :

[0039]

[0040] The present invention realizes super-resolution reconstruction in relatively low-resolution images. First, instance information in the image is added, and spatial position information is extracted through a spatial feature modulator, and then added to a feedback reconstruction subnet including a projection module for cyclic reconstruction. Adding instance information modulated by the spatial feature modulator and using a recurrent neural network will not only better reconstruct low-resolution images, but also have a significant effect on instance texture information, greatly improving the effect of image reconstruction.

[0041] Beneficial Effects

[0042] First, the present invention introduces instance space features into the super-resolution reconstruction of images, designs an instance space feature modulator, extracts instance space features, and modulates the super-resolution reconstruction features of images; secondly, designs a feedback reconstruction subnet, based on a loop feedback network, makes full use of high-frequency information and texture information, further uses a projection module, and introduces a residual fusion method, so that while improving the reconstruction effect, the training efficiency of the model can be guaranteed, and the scalability of image super-resolution reconstruction can be further improved. This method is stable and efficient, and has achieved good reconstruction results on a variety of public image super-resolution reconstruction datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is an overall flow chart of the image super-resolution reconstruction method based on instance space feature modulation and feedback mechanism of the present invention;

[0044] Figure 2 : is a network structure diagram of an image super-resolution reconstruction method based on instance space feature modulation and feedback mechanism of the present invention;

[0045] Figure 3 is a network structure diagram of the feedback reconstruction subnet in the present invention;

[0046] Figure 4 It is a specific network structure diagram of the projection module in the present invention. DETAILED DESCRIPTION

[0047] The present invention proposes an image super-resolution reconstruction method based on instance space feature modulation and feedback mechanism. Different from the previous method that only uses low-resolution images as the input of the reconstruction model, the present invention introduces the instance segmentation map of the low-resolution image, and uses it and the low-resolution image as the input of the reconstruction model at the same time. First, the instance space feature modulator extracts the instance space features of the low-resolution image based on the instance segmentation map, and modulates the super-resolution reconstruction features of the image based on the extracted instance space features; then, the low-resolution image reconstruction features after instance space feature modulation are used as the input of the feedback reconstruction subnet, and a reconstruction result is obtained through up / down sampling operations of different scales in the projection module, and the reconstruction result is iteratively reconstructed and optimized multiple times based on the loop feedback mechanism to obtain the final reconstruction result.

[0048] The present invention is described in more detail below with reference to specific examples and drawings.

[0049] The present invention comprises the following steps:

[0050] 1) First, obtain the training set and test set images of the COCO-2017 dataset, a total of 117,040 images and 25,000,000 instance labels, as well as the corresponding instance annotation information. Use the instance annotation information to create a mask corresponding to the instance segmentation map, and use the created mask to obtain the corresponding complete instance segmentation map from the image.

[0051] 2) Preprocess the dataset. For a dataset training image of size H×W, the current initial image size is H×W×3. First, resize it to 640×480×3, and perform data enhancement by random erasing, random flipping, and random cropping. Then, downsample the convolution kernel of size 1×1 to obtain the initial super-resolution reconstruction feature of size 256×256×3.

[0052] 3) The instance space feature modulator convolves the input instance segmentation map to extract the instance space features of the image; then, a two-dimensional Gaussian distribution is used to encode the extracted instance space features to generate an instance space heat map HP of size 256×256×3.

[0053] 4) Then, the instance space heat map HP containing the instance space position information is point-multiplied with the initial super-resolution reconstruction feature to modulate the initial super-resolution reconstruction feature and strengthen the spatial position information of the instance in the reconstructed feature; and after 3×3 downsampling, a feature of size 64×64×3 is obtained as the input F of the feedback reconstruction subnet in .

[0054] 5) The input of the feedback reconstruction subnet is the underlying image feature F modulated by the instance space feature in , which obtains a reconstruction result by feedback reconstruction subnet. Specifically, for the tth iteration of feedback reconstruction subnet:

[0055] 5.1) The input of the tth iteration of the feedback reconstruction subnet is: the reconstruction feature F fused with the instance space feature in The output of the previous iteration of the projection module in the feedback reconstruction subnetwork

[0056] 5.2) F in and Perform concat operation;

[0057] 5.3) Convolve the features after the concat operation to obtain H t , then, H t Obtain high-resolution features through deconvolution

[0058] 5.4) The low-resolution features are obtained through 3×3 convolution and combined with H t Subtract to obtain residual features Then the residual feature Convolution and deconvolution are performed sequentially and combined with high-resolution features Fusion, get the fused features

[0059] 5.5) Repeat the feature residual-fusion operation in 5.4) twice;

[0060] 5.6) Finally, we get the refined high-level features As the output of the projection module for the tth time

[0061] 5.7) and F in Together they serve as the input of the feedback reconstruction subnetwork for the next reconstruction iteration. After multiple iterations of optimization, the final reconstruction result is obtained.

[0062] 5.8) At the same time, the output of the projection module in each iteration After two upsampling operations, the super-resolution reconstruction result SR of the tth iteration is obtained. t , participate in the loss calculation of the model.

[0063] 6) The present invention uses the pixel-by-pixel mean square error commonly used in super-resolution reconstruction as the loss function. At the same time, in order to make the hidden states in the feedback reconstruction subnet contain information about the high-resolution image, each iterative reconstruction result of the model participates in the calculation of the loss function value. The loss function of the tth iteration of the present invention is shown in formula (9):

[0064]

[0065] Where t is the number of iterations, is the reconstructed image obtained at the tth iteration, I HR is the corresponding high-resolution image, m and n represent the width and height of the image respectively, α t represents the weight of the tth iteration.

[0066] 7) The network model of the present invention was trained on an Intel(R) Core i7-8700K@3.70GHz CPU, NVIDIA GeForceGTX 1080Ti GPU, Ubuntu 16.04 64-bit operating system and python3.6.2 development environment based on the Pytorch open source framework at two magnification scales of ×2 and ×4 for a total of 1000 epochs. At the beginning of training, the learning rate was set to 1e-4, and the learning rate was reduced to one-tenth of the original after every 200 epochs.

[0067] The present invention provides an image super-resolution reconstruction method based on instance space feature modulation and feedback mechanism, which is suitable for reconstructing low-resolution images into clear high-resolution images. According to experimental results, the present invention has good robustness, stable effect, high accuracy, and is an image super-resolution reconstruction method with stable effect and good training efficiency.

Claims

1. An image super-resolution reconstruction method based on instance space feature modulation and feedback mechanism, characterized in that: In the super-resolution reconstruction of low-resolution images, the instance space features in the image are introduced, the super-resolution reconstruction features are modulated by the instance space feature modulation module, and then input into the feedback reconstruction subnet including the projection module for cyclic reconstruction of the super-resolution reconstruction; 1) Input a low-resolution image and extract features as the initial super-resolution reconstruction features; 2) Design an instance space feature modulator, and use the instance information in the instance segmentation map corresponding to the input low-resolution image as the prior knowledge for super-resolution reconstruction; the instance space feature modulator extracts the instance space features of the low-resolution image, and modulates the initial super-resolution reconstruction features by the dot product to obtain the modulated image super-resolution reconstruction features; by modulating the initial super-resolution reconstruction features by the instance space features; 3) Based on the loop feedback mechanism, a feedback reconstruction subnet is designed and a projection module is introduced; The modulated image super-resolution reconstruction features are input into the feedback reconstruction subnet; a reconstruction result is obtained through up / down sampling operations of different scales in the projection module, and the reconstruction result is iteratively optimized multiple times based on the loop feedback mechanism to obtain the final reconstruction result; The instance space feature modulator in step 2) obtains the instance space feature, specifically: 2.1) For the image super-resolution reconstruction task, the spatial information of the image is very important; the instance space feature modulator extracts the instance space features of the instance segmentation map, and modulates the initial super-resolution reconstruction features based on the extracted instance space features to enhance the influence of the instance space features in the super-resolution reconstruction process; The specific instance spatial feature modulator consists of a convolutional layer and a two-dimensional Gaussian distribution layer; First, the instance space feature modulator performs a 3×3 convolution on the input instance segmentation map to extract the instance space features of the image; then, a two-dimensional Gaussian distribution is used to encode the extracted instance space features to generate an instance space heat map HP: in, Represents the instance segmentation map corresponding to the low-resolution image, Conv 3×3 represents a 3×3 convolution operation, and Gau represents a two-dimensional Gaussian distribution; 2.2) Based on the instance space heat map HP containing instance space feature information, the initial super-resolution reconstruction feature is modulated to enhance the instance space feature information in the super-resolution reconstruction feature. After downsampling, the modulated super-resolution reconstruction feature of size 64×64×3 is obtained as the input F of the feedback reconstruction subnet. in : F in =f(x·HP) (2) Where HP is the instance space heat map containing instance space feature information, x is the initial super-resolution reconstruction feature obtained by feature extraction of the low-resolution image, · represents the multiplication of the corresponding elements of the matrix, and f represents two 3×3 convolution operations; F in It is the super-resolution reconstruction feature after instance space feature modulation, which not only contains the low-resolution image features, but also has the features of the instance segmentation map and instance space feature information through the instance space feature modulator; In step 3), a feedback reconstruction subnet is designed to perform multiple iterative reconstruction optimizations on the super-resolution reconstruction features after the instance space feature modulation; The processing of the tth iteration of the feedback reconstruction subnet is as follows: 3.1) The super-resolution reconstruction feature F modulated by the instance space feature in As the input of the feedback reconstruction subnet, after being processed by the feedback reconstruction subnet, the output is obtained in, is the output of the t-1th iteration feedback reconstruction subnetwork, F in is the super-resolution reconstruction feature modulated by the instance space feature, P t It is the projection module in the feedback reconstruction subnetwork; Among them, the super-resolution reconstruction feature F in And the output of the t-1th iteration feedback reconstruction subnetwork As the input of the projection module, it is processed as follows; 3.1.1) First, F in and Perform concat operation and then obtain H after 1×1 convolution t : 3.1.2) Then, H t Obtain high-resolution features through deconvolution The low-resolution features are obtained through 3×3 convolution and combined with H t Subtract to obtain residual features Residual features Contains some feature information that is not shared by high-level features and low-level features; 3.1.3) Then the residual feature After 1×1 convolution, deconvolution is performed and combined with high-resolution features Fusion, get the fused features 3.1.4) The projection module repeats the feature residual-fusion operation of 3.1.1), 3.1.2) and 3.1.3) three times, and finally obtains the refined high-level features That is, as the output of the projection module for the tth time By performing residual-fusion operation on the features, the output Obtain compensation information at different scales; 3.2) After two upsampling operations, the super-resolution reconstruction result SR of the tth iteration is obtained. t :

2. The image super-resolution reconstruction method based on instance space feature modulation and feedback mechanism according to claim 1, characterized in that: In step 1), a low-resolution image is input and image features are extracted, specifically: 1) First, the input of the model is a low-resolution image and its corresponding instance segmentation map. The low-resolution image is passed through a 1×1 convolutional layer to obtain a 256×256×3 feature map as the initial super-resolution reconstruction feature.

Images