Reference map based multi-match search and super-resolution reconstruction method

Abstract

The application provides a multi-match search and super-resolution reconstruction method based on a reference image, a multi-match search and super-resolution reconstruction network based on a reference image is used as a deep network for super-resolution reconstruction based on a reference image, a low-resolution image is input into the multi-match search and super-resolution reconstruction network based on the reference image, and a high-resolution image is output by the reconstruction network. The method can reconstruct a high-resolution image with better texture details.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a multi-matching search and super-resolution reconstruction method based on reference images. BACKGROUND

[0002] Reference image based super-resolution reconstruction (RefSR) reconstructs more accurate image details from input low-resolution images by referencing additional reference images. Reference images usually have similar content to LR images and can provide real details. In recent years, most related research works have adopted CNN-based frameworks. Part of the research of RefSR is to implicitly align the Ref image and the LR image through the network, and CrossNet estimates the optical flow between the Ref image and the LR image at multiple scales and uses the optical flow to change the Ref image features. However, it relies on the optical flow estimation network, resulting in high computational burden, inaccurate estimation, and other defects. Shim et al. further proposed to align and extract Ref image features by using deformable convolution, and these alignment-based methods are limited in finding long-distance correspondences.

[0003] Another part of the research of RefSR is based on the idea of block matching. Zheng et al. trained two networks to learn feature correspondence and block synthesis, respectively. SRNTT performs multi-level image block matching between Ref image features and LR image features extracted from a pre-trained VGG network, and fuses the exchanged Ref image features and LR image features together to generate super-resolution reconstruction results. TTSR further introduces the transformer architecture into the RefSR task, and the hard attention and soft attention in the transformer help to more accurately transfer texture features from the Ref image.

[0004] However, the block matching method of SRNTT and TTSR has high computational cost. And it uses VGG network as a feature extractor, which has a large network size and needs to be pre-trained. SRNTT and TTSR both apply an enumeration block matching mechanism on feature maps, and then exchange relevant texture features. Since both use an enumeration method, it results in high computational cost. MASA adopts a hierarchical enumeration correspondence matching method, which significantly reduces the computational cost while achieving strong matching and migration capabilities, but still does not fully utilize the high-frequency details in the reference image. Therefore, implementing multi-matching of the reference image can obtain more texture information. SUMMARY

[0005] In order to overcome the defects and deficiencies existing in the prior art, the present application aims to provide a multi-match search and super-resolution reconstruction method based on a reference image, a multi-match search and super-resolution reconstruction network based on a reference image is used as a deep network for super-resolution reconstruction based on a reference image, a low-resolution image is input into the multi-match search and super-resolution reconstruction network based on a reference image, and a high-resolution image is output by the reconstruction network. The method can reconstruct a high-resolution image with better texture details.

[0006] The present application specifically adopts the following technical solutions:

[0007] A multi-match search and super-resolution reconstruction method based on a reference image, characterized in that a multi-match search and super-resolution reconstruction network based on a reference image is used as a deep network for super-resolution reconstruction based on a reference image, a low-resolution image I LR is input into the multi-match search and super-resolution reconstruction network based on a reference image, and a high-resolution image I SR is output by the reconstruction network.

[0008] Further, the multi-match search and super-resolution reconstruction network based on a reference image comprises an encoding module, a multi-match search and extraction module, and a decoding module; the encoding module is used to extract the features of a low-resolution image, a reference image, and a bicubic down-sampled reference image, then the multi-match search and extraction module is used to search for multiple image blocks similar to the low-resolution image on the bicubic down-sampled reference image, and extract corresponding image blocks on a high-resolution reference image feature map to obtain a new reference image through weighted summation; finally, the decoding module is used to reconstruct a high-resolution image I LR from the multi-scale fusion of the low-resolution image I SR .

[0009] Further, the multi-match search and extraction module comprises a coarse matching stage, a fine matching stage, and a feature extraction stage; the fine matching stage is used to calculate the weight corresponding to each reference image block and the similarity between the new reference image block and the low-resolution image block.

[0010] Further, in the first stage, the multi-match search and extraction module expands the LR feature map into multiple image blocks, for each LR image block, the cosine similarity between the center of the obtained LR image block and each reference image block is calculated to preliminarily search for the bicubic down-sampled reference image block most relevant to it; then multiple dx×dy size blocks are cut around the image block similar to the LR image in the reference image; and the reference image blocks at the corresponding positions on the reference image feature maps of different scales are cut, and the cut reference image blocks are used for fine matching in the second stage.

[0011] In the second stage, using the coarse matching of the first stage to obtain the corresponding multiple reference image blocks Block of the LR image block patch, matching the 3x3 patch of the LR image block patch and the corresponding multiple reference image blocks Block and calculating the cosine similarity; in order to obtain the corresponding relationship between the multiple reference image blocks and the LR image block, using the unique solution of the linear equation to obtain the weight corresponding to each reference block, and then weighting the multiple reference image blocks to obtain a new reference image block similar to the LR image block; and calculating the similarity between the low resolution block and the new reference image block; the coordinates, weights and cosine similarity corresponding to the reference image block obtained in the second stage are input to the next stage;

[0012] In the third stage, according to the coordinates obtained in the second stage, the corresponding positions are found in the reference image feature map F ref The corresponding positions are found in the reference image feature map F

[0013] Further, the new warp map is input into the decoding module in the last stage of the third stage of the multi-match search and extraction module, and the SR image with high frequency details is reconstructed by multi-scale fusion with the LR feature map.

[0014] Further, in the second stage of the multi-match search and extraction module, a single LR image block is matched with multiple reference image blocks to more fully utilize the high frequency information of the reference image blocks.

[0015] Further, when a set of training data (I LR ,I HR ) is provided, wherein I LR is a low resolution image in the training set, and I HR is a high resolution image in the training set, the loss function is represented as:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] wherein, Θ is a set of network parameters, I SR is I LR an output image obtained by inputting the low-resolution image to the multi-match search and super-resolution reconstruction network of the reference image; training the multi-match search and super-resolution reconstruction network of the reference image with the loss function as an objective function of the training process; and obtaining network parameters of the multi-match search and super-resolution reconstruction network of the reference image after the training, and directly using the network parameters for image super-resolution reconstruction.

[0022] Further, the encoding module and the decoding module are composed of a plurality of convolutional layers and residual blocks.

[0023] A multi-match search and super-resolution reconstruction system based on a reference image, characterized by comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor, and when the processor executes the computer program instructions, the method described above can be implemented.

[0024] Compared with the prior art, the present application and the preferred scheme thereof provide a multi-match search and super-resolution reconstruction method based on a reference image, which significantly reduces the computational cost compared with the previous matching algorithm. The method extracts the features of a low-resolution image and a reference image through an encoding module, then uses a multi-match search and extraction module to perform multi-match search on the reference image to obtain a plurality of reference image blocks similar to the low-resolution image block, and obtains a new reference image by weighted summation of the plurality of reference image blocks, so that the high-resolution reference image can be fully utilized. The method can utilize more high-frequency details in the reference image, thereby obtaining a high-resolution image with better visual effect.

[0025] The method is an improvement on the matching acceleration and spatial adaptive method (MASA-SR) for super-resolution reconstruction based on a reference image. The framework of the method mainly consists of three parts: an encoder module, a matching and extraction module, and a decoder module. High-quality corresponding matching is crucial, and high computational efficiency is also required. The method designs a mechanism for corresponding matching of a single low-resolution image block with a plurality of reference image blocks and weighted summation of the plurality of reference image blocks to obtain a new reference image. This design can utilize high-frequency information in the reference image to a greater extent. Experimental results show that the improved method is superior to the original MASA-SR method in terms of quantitative and qualitative evaluation of pictures. BRIEF DESCRIPTION OF DRAWINGS

[0026] The present application will be further described in detail below in combination with the drawings and specific embodiments:

[0027] Figure 1 is a structural schematic diagram of the multi-match search and super-resolution reconstruction network of the reference image in the embodiment of the present application.

[0028] Figure 2 is a comparison chart of the reconstruction results of the method and the other two methods in the embodiments of the present application. DETAILED DESCRIPTION

[0029] In order to make the features and advantages of the patent more obvious and easy to understand, the following specific examples are described in detail below, and the drawings are described as follows:

[0030] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0031] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0032] The present embodiment provides a reference image-based multi-match search and super-resolution reconstruction method, taking a reference image-based multi-match search and super-resolution reconstruction network as a deep network for reference image-based super-resolution reconstruction, and inputting a low-resolution image I LR to the reference image-based multi-match search and super-resolution reconstruction network, and the reconstruction network outputs a high-resolution image I SR .

[0033] Specifically, the reference image-based multi-match search and super-resolution reconstruction network of the present embodiment comprises an encoding module, a multi-match search and extraction module, and a decoding module.

[0034] The encoding module extracts the features of the low-resolution image, the reference image, and the bicubic down-sampled reference image, and then uses the multi-match search and extraction module to search for multiple image blocks similar to the low-resolution image on the bicubic down-sampled reference image, and extracts the corresponding image blocks on the high-resolution reference image feature map to obtain a new reference image by weighted summation; finally, the low-resolution image I LR and the new reference image are multi-scale fused by the decoding module to reconstruct a high-resolution image I SR . The encoding module and the decoding module are composed of a plurality of convolution layers and residual blocks.

[0035] As Figure 1As shown, in this embodiment, the multi-match search and extraction module is divided into three stages: in the first stage, the LR feature map is expanded into multiple image blocks (blocks), and for each LR image block, a plurality of bicubic down-sampled reference image blocks most relevant to the LR image block are preliminarily searched. The most relevant reference image block is found by calculating the cosine similarity between the center of the obtained LR image block and each reference image block, and then a plurality of dx x dy size blocks are cut around the image block similar to the LR image block in the reference image; according to the locality principle, the image block most similar to the LR image is likely to be in this block. At the same time, the reference image blocks at the corresponding positions on the reference image feature maps of different scales are also cut, and the cut reference image blocks are used for fine matching in the second stage.

[0036] In the second stage, the LR image blocks (patches) corresponding to the plurality of Ref image blocks (blocks) obtained in the first stage are matched and the cosine similarity is calculated for each 3x3 patch in the LR image block patch and the plurality of Ref image blocks Block. In order to make more full use of the high-frequency details of other image blocks, a single LR image block is used to correspond to a plurality of reference image blocks in this embodiment. In order to obtain the corresponding relationship between the plurality of reference image blocks and the LR image block, a unique solution linear equation set is used to obtain the weight corresponding to each reference block. Since it is difficult to obtain an equation, a plurality of reference image blocks are weighted and summed to obtain a new reference image block similar to the LR image block, and the similarity between the low-resolution block and the new reference image block is calculated. The coordinates, weights and cosine similarity of the reference image blocks obtained in the second stage are input to the next stage.

[0037] In the third stage, the corresponding positions are found in the reference image feature map F ref and extracted according to the coordinates obtained in the second stage, and then a plurality of reference image blocks with high-frequency details are obtained by using weighted summation on the plurality of reference image blocks, wherein the weights obtained in the second stage are used. Then, the reference image blocks corresponding to each LR image block are migrated and spliced in the order of the LR image blocks to obtain a warp image. Since the reference image features with higher cosine similarity are more useful, the warp image is multiplied by the cosine similarity to obtain a new warp image. After obtaining the new warp image, it is folded back. Finally, the new warp image is input into the decoding module to perform multi-scale fusion with the LR feature map to reconstruct an SR image with high-frequency details.

[0038] The multi-match search and extraction module enables a single LR image block to correspond to a plurality of reference image blocks, which makes more full use of the high-frequency information of the reference image blocks and improves the visual quality of the texture region.

[0039] When a set of training data sets (I LR ,I HR ) is provided, wherein I LR is a low-resolution image in the training set, I HR is a high-resolution image in the training set, the loss function thereof is represented as:

[0040]

[0041]

[0042]

[0043]

[0044]

[0045] Wherein, Θ is a set of network parameters, I SR is an output image of the input I LR to the reference map multi-match search and super-resolution reconstruction network; the loss function is used as the objective function of the training process, and the reference map multi-match search and super-resolution reconstruction network is trained; after the training is completed, the network parameters of the reference map multi-match search and super-resolution reconstruction network are obtained, which are directly used for image super-resolution reconstruction.

[0046] The application also provides a reference map multi-match search and super-resolution reconstruction system based on the reference map, comprising a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor, when the processor executes the computer program instructions, the above-mentioned method steps can be realized.

[0047] In the embodiment, 11871 training images in the CUFED5 data set are used to train the MMS. Data enhancement is realized by rotating 90, 180 and 270 degrees and horizontal flipping. The encoder has 3 building blocks, each block is composed of 1 convolution layer and 4 residual blocks. The fusion module contains a spatial adaptive module and a double residual module, which are composed of multiple convolution layers and residual blocks. In the multiscale module of 1x, 2x and 4x, the residual block numbers corresponding to 12, 8 and 4 are respectively. The number of all intermediate channels is 64. The activation function uses ReLU, and no batch layer (BN) is used in the network. The scale factor of the model is ×4.

[0048] In the multi-match search and extraction module, the size of the LR image block is 10x10, the size of the center patch used for matching is set to 3x3, and the size of the block corresponding to the patch of the LR image is set to 15x15. One LR image block corresponds to topk Ref image blocks, and topk is set to 3. The model is trained using the Adam optimizer, and beta1 and beta2 are set to 0.9 and 0.999, respectively. The learning rate is set to 1e-4, and the batch size is set to 10. The weight coefficients λ rec , λ per , and λ adv in the loss function are set to 1, 1, and 5e-3, respectively. In order to test, three benchmark datasets are used in this embodiment: CUFED5, Sun80, and Urban100. The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used to measure the quality of the SR image.

[0049] In order to be able to make a fair comparison with the MSE minimization method based on PSNR and SSIM, this embodiment trains a model with only a reconstruction loss function, which is called MMS_rec. This method is compared with eight representative deep learning-based single-image super-resolution reconstruction methods and four deep learning-based reference image super-resolution reconstruction methods. The twelve methods are SRCNN, MDSR, RDN, RCAN, HAN, SRGAN, ENet, ESRGAN, and CrossNet, SRNTT, TTSR, and MASA.

[0050] The quantitative effect of the method of the present application compared with the above-mentioned twelve methods is shown in Table 1, and the evaluation indexes are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), and the values are the average of the reconstruction results of the test data sets, wherein the unit of PSNR is dB.

[0051] Table 1

[0052]

[0053]

[0054] The experimental data show that the method of the present application has better performance than other super-resolution reconstruction methods, and is superior to other methods in terms of PSNR and SSIM evaluation indexes.

[0055] In terms of visual reconstruction effect, the super-resolution reconstruction effects of the two comparison methods and the method of the present application are compared as shown in Figure 2 It can be seen from Figure 2 that the multi-match search and super-resolution reconstruction method based on reference images of the present application can better reconstruct more accurate texture details.

[0056] It can be seen that the MMS of the present application significantly reduces the calculation cost compared with the previous matching algorithm by the multi-match search and extraction module, and makes full use of the high-resolution reference image. The multi-match search mechanism can utilize more high-frequency details in the reference image, thereby obtaining a high-resolution image with better visual effect.

[0057] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0058] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0059] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0060] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0061] The patent is not limited to the above best mode, anyone can draw other various forms of reference image-based multi-match search and super-resolution reconstruction method under the inspiration of the patent, any equivalent changes and modifications made in the scope of the patent application should be covered by the patent.

Claims

1. A method for multi-matching search and super-resolution reconstruction based on a reference graph, characterized in that: Using a multi-match search and super-resolution reconstruction network based on a reference image as a deep network for super-resolution reconstruction of the reference image, the low-resolution image is reconstructed. The input is a multi-match search and super-resolution reconstruction network to a reference image, and the reconstruction network outputs a high-resolution image. ; The multi-match search and super-resolution reconstruction network for the reference graph includes: an encoding module, a multi-match search and extraction module, and a decoding module; The encoding module extracts features from the low-resolution image, the reference image, and the bicubic downsampled reference image. Then, a multi-match search and extraction module searches for multiple image patches similar to the low-resolution image on the bicubic downsampled reference image, and extracts corresponding image patches from the feature map of the high-resolution reference image, performing a weighted summation to obtain a new reference map. Finally, the decoding module decodes the low-resolution image... High-resolution images were reconstructed by multi-scale fusion with new reference maps. .

2. The multi-match search and super-resolution reconstruction method based on a reference graph according to claim 1, characterized in that: The multi-match search and extraction module includes: a coarse matching stage, a fine matching stage, and a feature extraction stage; the fine matching stage is used to calculate the weight corresponding to each reference image block and the similarity between the new reference image block and the low-resolution image block.

3. The multi-match search and super-resolution reconstruction method based on a reference graph according to claim 2, characterized in that: In the first stage, the multi-match search and extraction module expands the LR feature map into multiple image blocks. For each LR image block, the cosine similarity between the center of the obtained LR image block and each reference image block is calculated to initially search for the most relevant reference image block after bicubic downsampling. Subsequently, multiple dx×dy-sized blocks are cropped around image patches similar to the LR image in the reference image; and reference image blocks at corresponding positions are cropped on the feature maps of the reference images at different scales. The cropped reference image blocks are used for fine matching in the second stage. In the second stage, the coarse matching from the first stage is used to obtain multiple reference image blocks corresponding to the LR image patch. The 3×3 patches in the LR image patch and the corresponding multiple reference image blocks are matched and the cosine similarity is calculated. In order to obtain the correspondence between multiple reference image blocks and LR image blocks, a system of linear equations with unique solutions is used to obtain the weight corresponding to each reference image block. Then, the multiple reference image blocks are weighted and summed to obtain a new reference image block that is similar to the LR image block. And calculate the similarity between the low-resolution patch and the new reference image patch; The coordinates, weights, and cosine similarity of the reference image patch obtained in the second stage are input into the next stage. In the third stage, the coordinates obtained in the second stage are used in the reference image feature map F. ref The corresponding position is found and extracted. Then, multiple reference image blocks are weighted and summed to obtain a reference image block with high-frequency details, using the weights obtained in the second stage. Subsequently, the reference image blocks corresponding to each LR image block are migrated and stitched together in the order of the LR image blocks to obtain a warp image. The warp image is multiplied by the cosine similarity to obtain a new warp image. After obtaining the new warp image, it is folded back.

4. The multi-match search and super-resolution reconstruction method based on a reference graph according to claim 3, characterized in that: At the end of the third stage of the multi-match search and extraction module, the new warp image is input into the decoding module and fused with the LR feature map at multiple scales to reconstruct an SR image with high-frequency details.

5. The multi-match search and super-resolution reconstruction method based on a reference graph according to claim 3, characterized in that: In the second stage of the multi-match search and extraction module, a single LR image block is matched with multiple reference image blocks to make fuller use of the high-frequency information of the reference image blocks.

6. The multi-match search and super-resolution reconstruction method based on a reference graph according to claim 1, characterized in that: When a set of training datasets is provided ,in For low-resolution images in the training set, For high-resolution images in the training set, the loss function is expressed as: in, It is a set of network parameters. yes The output image is input into the multi-match search and super-resolution reconstruction network of the reference image; the loss function is used as the objective function for the training process to train the multi-match search and super-resolution reconstruction network of the reference image; after training, the network parameters of the multi-match search and super-resolution reconstruction network of the reference image are obtained and directly used for image super-resolution reconstruction.

7. The multi-match search and super-resolution reconstruction method based on a reference graph according to claim 1, characterized in that: The encoding and decoding modules consist of multiple convolutional layers and residual blocks.

8. A multi-matching search and super-resolution reconstruction system based on a reference graph, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the method as described in any one of claims 1-7.

Images