A multi-frame image arbitrary scale super-resolution reconstruction method

Abstract

The application provides a multi-frame image arbitrary scale super-resolution reconstruction method, and specifically comprises the following steps: acquiring multiple continuous shooting images; performing alignment on the acquired multiple continuous shooting images by using a translation transformation in affine transformation; setting a target super-resolution magnification; constructing a super-resolution network, wherein the super-resolution network comprises a multi-frame feature fusion network, a multi-scale feature extraction network and an implicit feature reconstruction network; feeding the aligned multiple continuous shooting images into the super-resolution network to obtain a high-resolution image and output the high-resolution image. The application mines implicit features among pixels in fused features of multiple images, thereby increasing the utilization rate of sub-pixel information of multiple images, improving the reconstruction quality, and achieving the purpose of realizing arbitrary scale super-resolution reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to computer vision, and particularly relates to a multi-frame image arbitrary scale super-resolution reconstruction method. BACKGROUND

[0002] Super-resolution technology has always been a hot research topic in the field of computer vision. Super-resolution is divided into single-frame super-resolution and multi-frame super-resolution according to the number of low-resolution images used, but both are to reconstruct a high-resolution image with rich details from a low-resolution image. In single-frame super-resolution, only how to extract features and reconstruction needs to be concerned, while in multi-frame super-resolution, due to the sub-pixel microshift between multiple frames, the method of multi-frame image alignment, multi-frame image fusion and reconstruction needs to be considered.

[0003] At present, the research on super-resolution mainly focuses on single-frame super-resolution, and most of the methods are deep learning network methods, such as SRCNN, EDSR, SRGAN and RDN, etc. These methods have good high-resolution image reconstruction effect, but compared with multi-frame super-resolution, due to the lack of low-resolution image priori, it is difficult to reconstruct more detailed images.

[0004] In the research of multi-frame super-resolution, it can be generally divided into two categories, one is the method based on traditional calculation, and the other is the method based on deep learning. The method based on traditional calculation, such as POCS algorithm, although the calculation speed is fast, but it cannot well align and fuse multi-frame images, cannot fully utilize the sub-pixel information of multi-frame images, and the effect of reconstructing high-resolution images is poor. The method based on deep learning, such as DBSR, EBSR and BSRT, can well align and fuse multi-frame images and reconstruct high-resolution images with more details and better quality. However, the existing method based on deep learning cannot fully utilize the multi-level features extracted and fused, and the super-resolution multiple is fixed, one model can only realize one scale of super-resolution, if you want to realize multi-scale super-resolution, you need to repeat the model training. Multi-frame images have sufficient sub-pixel prior information, so they have the potential to realize arbitrary scale super-resolution, but the existing technology does not have a method to realize multi-scale super-resolution variable in multi-frame super-resolution.

[0005] Therefore, the present application proposes a multi-frame image arbitrary scale super-resolution reconstruction method to solve the above problems. SUMMARY

[0006] The purpose of the present application is to propose a multi-frame image arbitrary scale super-resolution reconstruction method, which excavates the implicit features between pixels in the fusion features of multi-frame images, thereby increasing the utilization rate of sub-pixel information of multi-frame images, improving the reconstruction quality, and achieving the purpose of realizing arbitrary scale super-resolution reconstruction.

[0007] To achieve the above object, the technical scheme of the present application is: a multi-frame image arbitrary scale super-resolution reconstruction method, specifically comprising the following steps:

[0008] S1: acquiring multiple continuous shooting images;

[0009] S2: aligning the acquired multiple continuous shooting images using translation transformation in affine transformation;

[0010] S3: setting a target super-resolution magnification;

[0011] S4: constructing a super-resolution network, the super-resolution network comprising a multi-frame feature fusion network, a multi-scale feature extraction network and an implicit feature reconstruction network;

[0012] S5: sending the aligned multiple continuous shooting images into the super-resolution network, first extracting a shallow feature map through a convolution layer, fusing features of the extracted shallow feature map through the multi-frame feature fusion network to obtain a fused feature map, obtaining a multi-scale feature map through the multi-scale feature extraction network, mining implicit features in the multi-scale feature map, inputting the implicit features after being spliced in the channel dimension into the implicit feature reconstruction network to obtain a pre-reconstruction image, performing bilinear interpolation magnification to a specified size on a first image or a specified reference image in the aligned multiple continuous shooting images, and then adding the image and the pre-reconstruction image pixel by pixel to obtain a high-resolution image and output the high-resolution image.

[0013] Preferably, the aligning of the acquired multiple continuous shooting images using translation transformation in affine transformation specifically comprises: selecting one image in the multiple continuous shooting images as a reference image, taking the reference image as a reference, calculating each affine matrix for translation transformation of each of the remaining images through an enhanced correlation coefficient maximization algorithm ECC, and performing translation transformation on each of the remaining images using the corresponding affine matrix to realize alignment with the reference image.

[0014] Preferably, the extraction of the shallow feature map through a convolution layer specifically comprises: extracting a shallow feature map of each image through a 3*3 convolution layer. Wherein, N represents the maximum number of frames of the multiple continuous shooting images.

[0015] Preferably, the fusing of features of the extracted shallow feature map through the multi-frame feature fusion network to obtain a fused feature map specifically comprises:

[0016] S5.1.1: taking a certain frame in the shallow feature map as a reference feature map, denoted as as the reference feature map;

[0017] S5.1.2: multiplying all the shallow feature maps by the reference feature map pixel by pixel to expand the difference between the feature maps, and the calculation process is as follows: wherein, denotes the expanded feature map and n = 1, …, N;

[0018] S5.1.3: Calculate the difference feature map R of each frame except the reference frame through the expanded feature map n , the calculation process is as follows:

[0019] S5.1.4: Calculate the attention fusion weight map for the current feature map through the difference feature map, the calculation process is as follows: A n = Sigmoid(R n ), n = 2, …, N, wherein, A n is the attention fusion weight map, R n is the difference feature map;

[0020] S5.1.5: Calculate the feature map for fusion of each frame except the reference frame using the attention fusion weight map, the calculation process is as follows: wherein F n is the feature map for fusion, is the initial shallow feature map;

[0021] S5.1.6: Concatenate all the obtained feature maps for fusion F n , n = 2, …, N and the initially selected reference feature map in the channel dimension, and input into the residual network to obtain the fusion feature map.

[0022] Preferably, the fusion feature map is used to obtain a multi-scale feature map through a multi-scale feature extraction network, specifically: first, the fusion feature map is projected to a deep feature space through a projection convolutional layer, then enters the down-sampling stage, and is extracted through the ST layer using the attention mechanism to obtain the highest-level feature, denoted as Then, through the down-sampling layer, the feature is down-sampled and then extracted through the ST layer to obtain the secondary-level feature Again, down-sampling is performed and the ST layer is used to obtain the low-level feature Then, enter the up-sampling stage, and After up-sampling, add pixel by pixel and then pass through the ST layer to obtain the secondary-level fusion feature of the first stage Again, up-sampling is performed and added pixel by pixel and then passed through the ST layer to obtain the highest-level fusion feature of the first stage Then, is again passed through the same down-sampling stage and up-sampling stage process to obtain the low-level feature of the second stage secondary-level fusion feature and high-level fusion feature Finally, the low-level features, the secondary fusion features and the highest-level fusion features obtained in the two stages are respectively spliced in the channel dimension, and three different output convolution projections are used to obtain three scales of deep fusion features Feat0, Feat1 and Feat2.

[0023] Preferably, the mining of the implicit features in the multi-scale feature map specifically comprises: obtaining normalized coordinates of each pixel point of a target high-resolution image according to a set target super-resolution magnification, linearly normalizing each pixel coordinate to a range of [-1, 1], and collecting an implicit feature vector corresponding to the coordinates in the multi-scale feature map according to the normalized coordinates.

[0024] Preferably, the implicit features are spliced in the channel dimension and then input into an implicit feature reconstruction network to obtain a pre-reconstruction image, specifically comprising: inputting the spliced implicit feature vectors into the implicit feature reconstruction network, passing through a series of MLP networks composed of full connection layers, Dropout layers and Relu activation function layers to obtain a feature vector for the reconstruction of each pixel, splicing all the feature vectors into a feature map according to the original coordinates, and finally outputting a pre-reconstruction image through a convolution layer.

[0025] Compared with the prior art, the present application has the following beneficial effects:

[0026] 1. The existing multi-frame super-resolution method only has a fixed magnification super-resolution, and the flexibility is insufficient and the large amount of low-resolution prior information cannot be fully utilized. The present application can fully utilize the prior information of the low-resolution, realize an arbitrary scale super-resolution method by mining the implicit features, and enhance the flexibility of the super-resolution model.

[0027] 2. Most of the existing multi-frame feature fusion methods directly stack all the features together and input them into the network for fusion. The multi-frame feature fusion in the present application pays more attention to the difference features between frames, supplements the insufficient information of the single-frame features through the difference features, and more fully fuses the multi-frame feature information.

[0028] 3. Most of the existing super-resolution methods are based on single-frame images, and it is difficult to reconstruct high-quality high-resolution images due to the lack of low-resolution prior information. The present application adopts the idea of multi-frame super-resolution, fully extracts and fuses the features of multiple frames of images, and reconstructs a super-resolution image of higher quality than the single-frame super-resolution. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The method flowchart of the present application;

[0030] Figure 2 The affine matrix diagram of the present application;

[0031] Figure 3 The overall schematic diagram of the super-resolution network of the present application is shown in the figure;

[0032] Figure 4 The schematic diagram of the multi-frame feature fusion network of the present application is shown in the figure;

[0033] Figure 5 The schematic diagram of the multi-scale feature extraction network of the present application is shown in the figure;

[0034] Figure 6 The schematic diagram of the implicit feature reconstruction network of the present application is shown in the figure. DETAILED DESCRIPTION

[0035] Most of the current super-resolution methods are for single-frame image super-resolution reconstruction, and the super-resolution reconstruction method based on single-frame image has less low-resolution priori information than the super-resolution method based on multi-frame image, and it is more difficult to reconstruct a high-quality high-resolution image. In the multi-frame based super-resolution reconstruction method, either some non-deep learning algorithms are used for reconstruction, but the reconstruction effect is general and often cannot cover many scenes, or only high-resolution images of fixed magnification can be reconstructed, and the sub-pixel information contained in the multi-frame images is not fully utilized. Therefore, the present application designs an arbitrary scale super-resolution reconstruction method based on multi-frame images, which excavates the implicit features between pixels in the fusion features of multi-frame images, thereby increasing the utilization rate of sub-pixel information of multi-frame images, improving the reconstruction quality, and achieving the purpose of realizing arbitrary scale super-resolution reconstruction.

[0036] The present application will be described in detail below with reference to the accompanying drawings Figures 1-6 The technical solutions of the present application will be described in detail.

[0037] As Figure 1 The present application provides an arbitrary scale super-resolution reconstruction method for multi-frame images, which specifically includes the following steps:

[0038] S1: acquiring multi-frame continuous shooting images;

[0039] S2: aligning the acquired multi-frame continuous shooting images by using the translation transformation in affine transformation;

[0040] S3: setting a target super-resolution magnification;

[0041] S4: constructing a super-resolution network, wherein the super-resolution network includes a multi-frame feature fusion network, a multi-scale feature extraction network and an implicit feature reconstruction network;

[0042] S5: the aligned multi-frame continuous shooting image is sent into the super-resolution network, shallow feature maps are extracted through a convolutional layer, the extracted shallow feature maps are subjected to feature fusion through a multi-frame feature fusion network to obtain fused feature maps, multi-scale feature maps are obtained through a multi-scale feature extraction network, implicit features are mined in the multi-scale feature maps, and the implicit features are input into an implicit feature reconstruction network after being spliced in the channel dimension to obtain a pre-reconstruction image; a first frame image or a specified reference image in the aligned multi-frame continuous shooting image is bilinearly interpolated and enlarged to a specified size, and then the image is pixel by pixel added to the pre-reconstruction image to obtain a high-resolution image and output.

[0043] In the embodiment, a series of continuous shooting images of the same scene in an extremely short time are first acquired, and the number of image frames is consistent with the number of frames used when the model is trained. At present, most of the image acquisition methods are to shoot the same scene by using a handheld camera to acquire a series of multi-frame images with a certain sub-pixel microshift by using the slight shaking of the hand. The image source of the present application is a series of images shot by using a professional camera by the above method, which has a series of advantages such as high authenticity, good continuity, and more sub-pixel information, and is beneficial to subsequent multi-frame reconstruction. In addition to directly inputting the image, the present application can also acquire more than 1 frame and less than 14 frames of images in an extremely short time from various video files with little change as input images.

[0044] In the embodiment, after the images are acquired, the multi-frame images are aligned to reduce the influence of excessive shaking on subsequent reconstruction; since the series of multi-frame images shot are images of the same focal length, only slight translation exists in the up, down, left and right directions, so translation transformation in affine transformation is used for alignment between images.

[0045] The translation transformation in affine transformation is used to align the acquired multi-frame continuous shooting images, one frame of image in the multi-frame continuous shooting images is selected as a reference image, each affine matrix for translation transformation of each of the remaining frames of images is calculated by using the enhanced correlation coefficient maximization algorithm ECC with the reference image as the reference, the affine matrix is as shown in Figure 2 , and each of the remaining frames of images is subjected to translation transformation by using the corresponding affine matrix to realize alignment with the reference image.

[0046] In the embodiment, after the images are aligned, the magnification multiple needs to be set for subsequent super-resolution reconstruction. Then all the aligned images need to be sent into the super-resolution network, as shown in Figure 3 , which is a whole network architecture diagram, feature fusion, multi-scale feature extraction, implicit feature mining and reconstruction are performed through the network, as shown in Figure 4 , 5 , 6, which are multi-frame feature fusion network, multi-scale feature extraction network, and implicit feature reconstruction network schematic diagrams.

[0047] In this embodiment, the shallow feature map extracted through a convolutional layer is specifically extracted through a 3*3 convolutional layer. Wherein, N represents the maximum frame number of the multi-frame continuous shooting images.

[0048] In this embodiment, the extracted shallow feature map is fused through a multi-frame feature fusion network to obtain a fused feature map, which is specifically:

[0049] S5.1.1: Take a frame in the shallow feature map as a reference feature map, denoted as , and the reference feature map is

[0050] S5.1.2: Multiply all the shallow feature maps by the reference feature map pixel by pixel to expand the difference between the feature maps, and the calculation process is as follows: Wherein, , n = 1, …, N, indicates the expanded feature map.

[0051] S5.1.3: Calculate the difference feature map R n except the reference frame through the expanded feature map, and the calculation process is as follows:

[0052] S5.1.4: Calculate the attention fusion weight map for the current feature map through the difference feature map, and the calculation process is as follows: n = Sigmoid(R n ), n = 2, …, N, wherein A n is the attention fusion weight map, and R n is the difference feature map.

[0053] S5.1.5: Calculate the feature map for fusion of each frame except the reference frame using the attention fusion weight map, and the calculation process is as follows: Wherein F n is the feature map for fusion, is the initial shallow feature map.

[0054] S5.1.6: Concatenate all the obtained feature maps F n for fusion, n = 2, …, N, and the initially selected reference feature map in the channel dimension, and input into the residual network to obtain the fused feature map.

[0055] In this embodiment, different levels of features are extracted through a multi-scale feature extraction network similar to the U-Net architecture, and three different scale feature maps are obtained through two levels of feature extraction and fusion.

[0056] 1. The multi-scale feature map is obtained by the multi-scale feature extraction network through the fusion feature map, specifically: first, the fusion feature map is projected to a deep feature space through a projection convolution layer, then enters the down-sampling stage, and the highest level feature is extracted through the ST layer using the attention mechanism, denoted as Then, through the down-sampling layer, the features are down-sampled and then the secondary features are obtained through the ST layer The down-sampling is performed again and the low-level features are obtained through the ST layer Then, the up-sampling stage is entered, and After up-sampling, and Pixel-by-pixel addition is performed and then the secondary fusion features of the first stage are obtained through the ST layer The up-sampling is performed again and Pixel-by-pixel addition is performed and then the highest level fusion features of the first stage are obtained through the ST layer Then, The low-level features of the second stage are obtained through the same down-sampling stage and up-sampling stage process again Secondary fusion features And high-level fusion features Finally, the low-level features, secondary fusion features, and highest level fusion features obtained in the two stages are respectively correspondingly spliced in the channel dimension, and three scales of deep fusion features Feat0, Feat1, and Feat2 are obtained through three different output convolution projections; wherein, the ST layer is a SwinTransformer layer.

[0057] In the embodiment, the implicit features are mined in the multi-scale feature map, specifically: according to the set target super-resolution magnification, the normalized coordinates of each pixel point of the target high-resolution image are obtained, each pixel coordinate is linearly normalized to the range of [-1, 1], and the implicit feature vector of the corresponding coordinate in the multi-scale feature map is collected according to the normalized coordinates.

[0058] In the embodiment, the implicit features are spliced in the channel dimension and then input into the implicit feature reconstruction network to obtain a pre-reconstruction image, specifically: the spliced implicit feature vector is input into the implicit feature reconstruction network, passes through a series of MLP networks composed of full connection layers, Dropout layers, and Relu activation function layers, and obtains the feature vector of each pixel for reconstruction, all feature vectors are spliced into a feature map according to the coordinates, and finally a pre-reconstruction image is output through a convolution layer.

[0059] The above is a preferred embodiment of the present application, and any changes made according to the technical solutions of the present application, as long as the generated function does not exceed the scope of the technical solutions of the present application, are within the protection scope of the present application.

Claims

1. A multi-frame image arbitrary scale super-resolution reconstruction method, characterized in that, The specific steps include: S1: Acquire multiple frames of continuous shooting images; S2: Align the acquired multi-frame continuous shooting images using the translation transformation in the affine transformation; S3: set the target super-resolution magnification; S4: constructing a super-resolution network, wherein the super-resolution network includes a multi-frame feature fusion network, a multi-scale feature extraction network, and an implicit feature reconstruction network; S5: The aligned multi-frame continuous shooting images are fed into the super-resolution network. A shallow feature map is first extracted through a convolutional layer. The extracted shallow feature map is fused through a multi-frame feature fusion network to obtain a fused feature map. The fused feature map is passed through a multi-scale feature extraction network to obtain a multi-scale feature map. Implicit features are mined in the multi-scale feature map, and the implicit features are spliced ​​in the channel dimension and input into the implicit feature reconstruction network to obtain a pre-reconstructed image. The first frame image in the aligned multi-frame continuous shooting images or the specified reference image is bilinearly interpolated and enlarged to a specified size. This image is then pixel-by-pixel added to the pre-reconstructed image to obtain a high-resolution image and output. The extracting shallow feature map through a convolution layer is specifically extracting a shallow feature map of each frame of image through a 3*3 convolution layer Wherein, N represents the maximum frame number of the multi-frame continuous shooting images. The extracted shallow feature map is subjected to feature fusion through a multi-frame feature fusion network to obtain a fused feature map, specifically: S5.1.1: Take a frame in the shallow feature map as a reference feature map, denoted as reference feature map; S5.1.2: Multiply all the shallow feature maps by the reference feature map pixel by pixel to expand the difference between the feature maps, the calculation process is as follows: wherein, denotes the expanded feature map and n = 1, …, N; S5.1.3: Calculate the difference feature map R excluding the reference frame from the enlarged feature map n The calculation process is as follows: S5.1.4: Calculate the attention fusion weight map for the current feature map through the difference feature map, and the calculation process is as follows: Wherein, A n is the attention fusion weight map, R n is the difference feature map; S5.1.5: Calculate the feature maps for fusion of each frame except the reference frame using the attention fusion weight map, the calculation process is as follows: n = 2, …, N, wherein F n is the feature map for fusion, is the initial shallow feature map; S5.1.6: all the feature maps F obtained for fusion are concatenated n , n = 2, …, N and the initially selected reference feature map In the channel dimension, the concatenated feature maps are input into a residual network to obtain the fused feature map; The fusion feature map is projected to a deep feature space through a projection convolution layer, and then enters a down-sampling stage, and the highest-level feature is extracted through an ST layer using an attention mechanism, denoted as Then, the feature is down-sampled through a down-sampling layer, and then the secondary-level feature is obtained through the ST layer The down-sampling is performed again, and the low-level feature is obtained through the ST layer Then, the up-sampling stage is entered, and the After up-sampling, the The pixel-by-pixel addition is performed, and then the secondary-level fusion feature of the first stage is obtained through the ST layer The up-sampling is performed again, and the The pixel-by-pixel addition is performed, and then the highest-level fusion feature of the first stage is obtained through the ST layer Then, the The low-level feature of the second stage is obtained through a same down-sampling stage and up-sampling stage process again The secondary-level fusion feature And the highest-level fusion feature Finally, the low-level features, the secondary-level fusion features, and the highest-level fusion features obtained in the two stages are respectively correspondingly spliced in the channel dimension, and three scales of deep fusion features Feat0, Feat1, and Feat2 are obtained through three different output convolution projections; wherein, the ST layer is a Swin Transformer layer.

2. The method of claim 1, wherein, The specific method of aligning the acquired multiple frames of continuous shooting images by using the translation transformation in the affine transformation is as follows: one frame of the multiple frames of continuous shooting images is selected as the reference image, and each affine matrix for the translation transformation of each of the remaining frames of the image is calculated based on the reference image through the enhanced correlation coefficient maximization algorithm ECC, and each of the remaining frames of the image is translated using the corresponding affine matrix to achieve alignment with the reference image.

3. The method of claim 1, wherein, The method of mining implicit features in the multi-scale feature map specifically comprises: obtaining the normalized coordinates of each pixel point in the target high-resolution image according to the set target super-resolution magnification, linearly normalizing each pixel coordinate to the range of [-1, 1], and collecting the implicit feature vector of the corresponding coordinate in the multi-scale feature map according to the normalized coordinate.

4. The method of claim 1, wherein, The implicit features are spliced ​​in the channel dimension and then input into the implicit feature reconstruction network to obtain a pre-reconstructed image. Specifically, the spliced ​​implicit feature vector is input into the implicit feature reconstruction network, and a feature vector of each pixel used for reconstruction is obtained through an MLP network composed of a fully connected layer, a Dropout layer, and a Relu activation function layer. All feature vectors are spliced ​​into a feature map according to the original coordinates, and finally the pre-reconstructed image is output through a convolutional layer.

Images