Multi-frame image arbitrary-scale super-resolution reconstruction method

Abstract

The invention provides a multi-frame image arbitrary-scale super-resolution reconstruction method. The method specifically comprises the following steps: acquiring multi-frame continuous shooting images; aligning the obtained multiple frames of continuous shooting images by adopting translation transformation in affine transformation; setting a target super-resolution magnification factor; constructing a super-resolution network, wherein the super-resolution network comprises a multi-frame feature fusion network, a multi-scale feature extraction network and an implicit feature reconstruction network; and sending the aligned multi-frame continuous shooting image into a super-resolution network to obtain a high-resolution image and outputting the high-resolution image. According to the method, the implicit features among the pixels are mined in the fusion features of the multi-frame image, so that the utilization rate of sub-pixel information of the multi-frame image is increased, the reconstruction quality is improved, and the purpose of realizing super-resolution reconstruction of any scale is achieved.

Description

Technical Field

[0001] The present invention relates to computer vision, and in particular to a method for reconstructing multiple-frame images in arbitrary scale super-resolution. Background Art

[0002] Super-resolution technology has always been a hot topic in the field of computer vision. Super-resolution is divided into single-frame super-resolution and multi-frame super-resolution according to the number of low-resolution images used, but both are aimed at reconstructing high-resolution images with rich details from low-resolution images. In single-frame super-resolution, we only need to focus on how to extract features and reconstruct them, while in multi-frame super-resolution, due to the sub-pixel shift between multiple frames, we need to consider the methods of aligning, fusing and reconstructing multiple frames.

[0003] At present, the research on super-resolution mainly focuses on single-frame super-resolution, and most of the methods use deep learning network methods, such as SRCNN, EDSR, SRGAN and RDN. These methods have good high-resolution image reconstruction effects, but compared with multi-frame super-resolution, due to the lack of low-resolution image priors, it is inherently difficult to reconstruct more detailed images.

[0004] In the research of multi-frame super-resolution, it can be generally divided into two categories, one is the method based on traditional computing, and the other is the method based on deep learning. The traditional computing-based methods, such as the POCS algorithm, have fast calculation speed, but they cannot align and fuse multiple frames of images well, cannot make full use of the sub-pixel information of multiple frames of images, and have poor effect in reconstructing high-resolution images. The deep learning-based methods, such as DBSR, EBSR and BSRT, can better align and fuse multiple frames of images and reconstruct high-resolution images with more details and better quality. However, the existing deep learning-based methods fail to make full use of the multi-level features extracted and fused, and the super-resolution multiple is fixed. A model can only achieve super-resolution of one scale. If you want to achieve multi-scale super-resolution, you need to repeatedly train the model. Multi-frame images have sufficient sub-pixel prior information and therefore have the potential to achieve super-resolution of any scale, but the existing technology does not have a method to achieve multi-scale super-resolution variables in multi-frame super-resolution.

[0005] Therefore, in view of the above problems, the present invention proposes a method for arbitrary-scale super-resolution reconstruction of multi-frame images to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to propose a method for super-resolution reconstruction of multiple images at any scale, by mining implicit features between pixels in the fusion features of multiple frames of images, thereby increasing the utilization rate of sub-pixel information of multiple frames of images, improving the reconstruction quality, and achieving the purpose of super-resolution reconstruction of any scale.

[0007] To achieve the above object, the technical solution of the present invention is: a method for super-resolution reconstruction of multiple frames of images at any scale, specifically comprising the following steps:

[0008] S1: Acquire multiple frames of continuous shooting images;

[0009] S2: using the translation transformation in the affine transformation to align the acquired multiple frames of continuous shooting images;

[0010] S3: Set the target super-resolution magnification;

[0011] S4: constructing a super-resolution network, wherein the super-resolution network includes a multi-frame feature fusion network, a multi-scale feature extraction network and an implicit feature reconstruction network;

[0012] S5: Send the aligned multi-frame continuous shooting images into the super-resolution network, first extract the shallow feature map through a convolution layer, fuse the extracted shallow feature map through a multi-frame feature fusion network to obtain a fused feature map, pass the fused feature map through a multi-scale feature extraction network to obtain a multi-scale feature map, mine implicit features in the multi-scale feature map, and input the implicit features into the implicit feature reconstruction network after splicing in the channel dimension to obtain a pre-reconstructed image, perform bilinear interpolation on the first frame image in the aligned multi-frame continuous shooting images or the specified reference image to enlarge it to a specified size, and then add this image to the pre-reconstructed image pixel by pixel to obtain a high-resolution image and output it.

[0013] Preferably, the use of translation transformation in affine transformation to align the acquired multiple frames of continuous shooting images is specifically as follows: one frame of the multiple frames of continuous shooting images is selected as a reference image, and each affine matrix for translation transformation of each remaining frame of the image is calculated through the enhanced correlation coefficient maximization algorithm ECC based on the reference image, and each remaining frame of the image is translated using the corresponding affine matrix to achieve alignment with the reference image.

[0014] Preferably, the extracting of shallow feature maps through a convolution layer is specifically: extracting shallow feature maps of each frame of image through a 3*3 convolution layer Wherein, N represents the maximum number of frames of the multi-frame continuous shooting image.

[0015] Preferably, the extracted shallow feature map is subjected to feature fusion through a multi-frame feature fusion network to obtain a fused feature map, specifically:

[0016] S5.1.1: Take a frame in the shallow feature map as the reference feature map, record is the reference feature map;

[0017] S5.1.2: Multiply all shallow feature maps pixel by pixel by the reference feature map to enlarge the difference between feature maps. The calculation process is as follows: in, represents the enlarged feature map and n=1,…,N;

[0018] S5.1.3: Calculate the difference feature map R other than the reference frame through the enlarged feature map n , the calculation process is as follows:

[0019] S5.1.4: Calculate the attention fusion weight map for the current feature map through the difference feature map. The calculation process is as follows: A n =Sigmoid(R n ), n = 2, ..., N, where A n is the attention fusion weight map, R n is the difference feature map;

[0020] S5.1.5: Use the attention fusion weight map to calculate the feature map for each frame except the reference frame. The calculation process is as follows: Among them, F n is the feature map used for fusion, is the initial shallow feature map;

[0021] S5.1.6: All feature maps F obtained for fusion n , n = 2, ..., N and the initially selected reference feature map Concatenate in the channel dimension and send it to the residual network to obtain the fused feature map.

[0022] Preferably, the fusion feature map is passed through a multi-scale feature extraction network to obtain a multi-scale feature map, specifically: the fusion feature map is first projected into a deep feature space through a projection convolution layer, and then enters the downsampling stage, and the highest-level features are extracted through the ST layer using the attention mechanism, which is recorded as Then, the features are downsampled through the downsampling layer and then passed through the ST layer to obtain secondary features. Downsample again and get low-level features through the ST layer Then enter the upsampling stage, After upsampling Add pixel by pixel and then pass through the ST layer to obtain the secondary fusion features of the first stage Upsample again with After pixel-by-pixel addition, the ST layer is used to obtain the highest-level fusion features of the first stage. Then Then go through the same downsampling and upsampling process to obtain the low-level features of the second stage Secondary fusion features and advanced fusion features Finally, the low-level features, secondary fusion features, and highest-level fusion features obtained in the two stages are concatenated in the channel dimension respectively, and three scales of deep fusion features Feat0, Feat1, and Feat2 are obtained through three different output convolution projections; wherein the ST layer is a SwinTransformer layer.

[0023] Preferably, mining implicit features in the multi-scale feature map specifically comprises: obtaining the normalized coordinates of each pixel point of the target high-resolution image according to the set target super-resolution magnification, linearly normalizing each pixel coordinate to the range of [-1,1], and collecting the implicit feature vector of the corresponding coordinate in the multi-scale feature map according to the normalized coordinate.

[0024] Preferably, the implicit features are spliced ​​in the channel dimension and then input into the implicit feature reconstruction network to obtain the pre-reconstructed image, specifically: the spliced ​​implicit feature vector is input into the implicit feature reconstruction network, and a feature vector of each pixel used for reconstruction is obtained through an MLP network composed of a series of fully connected layers, Dropout layers and Relu activation function layers, all feature vectors are spliced ​​into a feature map according to the original coordinates, and finally the pre-reconstructed image is output through a convolutional layer.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. The existing multi-frame super-resolution method only has a fixed magnification super-resolution, which is not flexible enough and fails to make full use of such a large amount of low-resolution prior information. The present invention can make full use of low-resolution prior information and realize a super-resolution method of any scale by mining implicit features, thereby enhancing the flexibility of the super-resolution model.

[0027] 2. Most of the existing multi-frame feature fusion methods directly stack all the features together and send them into the network for fusion. The multi-frame feature fusion in the present invention pays more attention to the difference features between frames, supplements the insufficient information of single-frame features through the difference features, and more fully fuses the multi-frame feature information.

[0028] 3. Most existing super-resolution methods are based on single-frame images. Due to the lack of low-resolution prior information, it is inherently difficult to reconstruct high-quality high-resolution images. The present invention adopts the idea of ​​multi-frame super-resolution to fully extract and fuse the features of multiple frame images to reconstruct a super-resolution image with higher quality than single-frame super-resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of the method of the present invention;

[0030] Figure 2 It is the affine matrix diagram of the present invention;

[0031] Figure 3 Schematic diagram of the overall architecture of the super-resolution network of the present invention;

[0032] Figure 4 This is a schematic diagram of a multi-frame feature fusion network of the present invention;

[0033] Figure 5 This is a schematic diagram of a multi-scale feature extraction network of the present invention;

[0034] Figure 6 This is a schematic diagram of the implicit feature reconstruction network of the present invention. DETAILED DESCRIPTION

[0035] At present, most super-resolution methods are for super-resolution reconstruction of single-frame images, and the super-resolution reconstruction method based on single-frame images inherently has less low-resolution prior information than the super-resolution method based on multi-frame images, and it is more difficult to reconstruct high-quality high-resolution images. In the super-resolution reconstruction method based on multi-frames, either reconstruction is performed through some non-deep learning algorithms, but the reconstruction effect is general and often cannot cover many scenes, or only high-resolution images with fixed magnifications can be reconstructed, and the sub-pixel information contained in the multi-frame images is not fully utilized. Therefore, the present invention designs an arbitrary scale super-resolution reconstruction method based on multi-frame images, which increases the utilization rate of sub-pixel information of multi-frame images, improves the reconstruction quality, and achieves the purpose of realizing arbitrary scale super-resolution reconstruction by mining implicit features between pixels in the fusion features of multi-frame images.

[0036] The following is combined with Figure 1-6 , the technical solution of the present invention is specifically described.

[0037] like Figure 1 The present invention proposes a method for super-resolution reconstruction of multiple frames of images at any scale, which specifically includes the following steps:

[0038] S1: Acquire multiple frames of continuous shooting images;

[0039] S2: using the translation transformation in the affine transformation to align the acquired multiple frames of continuous shooting images;

[0040] S3: Set the target super-resolution magnification;

[0041] S4: constructing a super-resolution network, wherein the super-resolution network includes a multi-frame feature fusion network, a multi-scale feature extraction network and an implicit feature reconstruction network;

[0042] S5: Send the aligned multi-frame continuous shooting images into the super-resolution network, first extract the shallow feature map through a convolution layer, fuse the extracted shallow feature map through a multi-frame feature fusion network to obtain a fused feature map, pass the fused feature map through a multi-scale feature extraction network to obtain a multi-scale feature map, mine implicit features in the multi-scale feature map, and input the implicit features into the implicit feature reconstruction network after splicing in the channel dimension to obtain a pre-reconstructed image, perform bilinear interpolation on the first frame image in the aligned multi-frame continuous shooting images or the specified reference image to enlarge it to a specified size, and then add this image to the pre-reconstructed image pixel by pixel to obtain a high-resolution image and output it.

[0043] In this implementation, it is first necessary to obtain a series of continuous images of the same scene in a very short time, and the number of image frames is consistent with the number of frames used when training the model. Currently, most image acquisition methods are to use a handheld camera to continuously shoot the same scene, thereby using the slight shaking of the hand itself to obtain a series of multi-frame images with certain sub-pixel micro-shifts. The image source of the present invention is a series of images taken by a professional camera using the above method, which has a series of advantages such as sufficient authenticity, good continuity, and a lot of sub-pixel information, which is conducive to subsequent multi-frame reconstruction. In addition to directly inputting images, the present invention can also obtain continuous images of more than 1 frame and less than 14 frames in a very short time as input images from various video files with little change.

[0044] In this embodiment, after acquiring the image, multiple frames of images need to be aligned to reduce the impact of excessive jitter on subsequent reconstruction; since the series of multiple frames of images taken are all images of the same focal length, there is only a slight translation up, down, left, and right, so the translation transformation in the affine transformation is used to align the images.

[0045] The method of aligning the acquired multiple frames of continuous shooting images by using the translation transformation in the affine transformation is as follows: one frame of the multiple frames of continuous shooting images is selected as a reference image, and each affine matrix of the translation transformation of each of the remaining frames of images is calculated by using the enhanced correlation coefficient maximization algorithm ECC. The affine matrix is ​​as follows: Figure 2 As shown, each of the remaining frames of images is translated using the corresponding affine matrix to achieve alignment with the reference image.

[0046] In this embodiment, after aligning the images, the magnification factor needs to be set for subsequent super-resolution reconstruction. Then, all the aligned images need to be sent to the super-resolution network, such as Figure 3 The figure shows the overall network architecture, which uses the network to perform feature fusion, multi-scale feature extraction, and implicit feature mining and reconstruction. Figure 4 , 5 , 6 show the schematic diagrams of the multi-frame feature fusion network, the multi-scale feature extraction network, and the implicit feature reconstruction network.

[0047] In this embodiment, the extraction of shallow feature maps through a convolution layer is specifically: extracting shallow feature maps of each frame image through a 3*3 convolution layer Wherein, N represents the maximum number of frames of the multi-frame continuous shooting image.

[0048] In this embodiment, the extracted shallow feature map is subjected to feature fusion through a multi-frame feature fusion network to obtain a fused feature map, specifically:

[0049] S5.1.1: Take a frame in the shallow feature map as the reference feature map, record is the reference feature map;

[0050] S5.1.2: Multiply all shallow feature maps pixel by pixel by the reference feature map to enlarge the difference between feature maps. The calculation process is as follows: in, represents the enlarged feature map and n=1,…,N;

[0051] S5.1.3: Calculate the difference feature map R other than the reference frame through the enlarged feature map n , the calculation process is as follows:

[0052] S5.1.4: Calculate the attention fusion weight map for the current feature map through the difference feature map. The calculation process is as follows: A n =Sigmoid(R n ), n = 2, ..., N, where A n is the attention fusion weight map, R n is the difference feature map;

[0053] S5.1.5: Use the attention fusion weight map to calculate the feature map for each frame except the reference frame. The calculation process is as follows: Among them, F n is the feature map used for fusion, is the initial shallow feature map;

[0054] S5.1.6: All feature maps F obtained for fusion n , n = 2, ..., N and the initially selected reference feature map Concatenate in the channel dimension and send it to the residual network to obtain the fused feature map.

[0055] In this embodiment, features at different levels are extracted through a multi-scale feature extraction network similar to the U-Net architecture, and feature maps of three different scales are obtained through two-level feature extraction and fusion.

[0056] 1. The fusion feature map is passed through a multi-scale feature extraction network to obtain a multi-scale feature map. Specifically, the fusion feature map is first projected into the deep feature space through a projection convolution layer, and then enters the downsampling stage, and the highest-level features are extracted through the ST layer using the attention mechanism, which is recorded as Then, the features are downsampled through the downsampling layer and then passed through the ST layer to obtain secondary features. Downsample again and get low-level features through the ST layer Then enter the upsampling stage, After upsampling Add pixel by pixel and then pass through the ST layer to obtain the secondary fusion features of the first stage Upsample again with After pixel-by-pixel addition, the ST layer is used to obtain the highest-level fusion features of the first stage. Then Then go through the same downsampling and upsampling process to obtain the low-level features of the second stage Secondary fusion features and advanced fusion features Finally, the low-level features, secondary fusion features, and highest-level fusion features obtained in the two stages are concatenated in the channel dimension respectively, and three scales of deep fusion features Feat0, Feat1, and Feat2 are obtained through three different output convolution projections; wherein the ST layer is a SwinTransformer layer.

[0057] In this embodiment, mining implicit features in a multi-scale feature map is specifically as follows: obtaining normalized coordinates of each pixel point of a target high-resolution image according to a set target super-resolution magnification factor, linearly normalizing each pixel coordinate to a range of [-1, 1], and collecting implicit feature vectors of corresponding coordinates in the multi-scale feature map according to the normalized coordinates.

[0058] In this embodiment, the implicit features are spliced ​​in the channel dimension and then input into the implicit feature reconstruction network to obtain the pre-reconstructed image. Specifically, the spliced ​​implicit feature vectors are input into the implicit feature reconstruction network, and a feature vector of each pixel used for reconstruction is obtained through a series of MLP networks composed of a fully connected layer, a Dropout layer, and a Relu activation function layer. All feature vectors are spliced ​​into a feature map according to the coordinates, and finally the pre-reconstructed image is output through a convolutional layer.

[0059] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A method for super-resolution reconstruction of multiple frames of images at any scale, characterized in that: The specific steps include: S1: Acquire multiple frames of continuous shooting images; S2: using the translation transformation in the affine transformation to align the acquired multiple frames of continuous shooting images; S3: Set the target super-resolution magnification; S4: constructing a super-resolution network, wherein the super-resolution network includes a multi-frame feature fusion network, a multi-scale feature extraction network and an implicit feature reconstruction network; S5: Send the aligned multi-frame continuous shooting images into the super-resolution network, first extract the shallow feature map through a convolution layer, fuse the extracted shallow feature map through a multi-frame feature fusion network to obtain a fused feature map, pass the fused feature map through a multi-scale feature extraction network to obtain a multi-scale feature map, mine implicit features in the multi-scale feature map, and input the implicit features into the implicit feature reconstruction network after splicing in the channel dimension to obtain a pre-reconstructed image, perform bilinear interpolation on the first frame image in the aligned multi-frame continuous shooting images or the specified reference image to enlarge it to a specified size, and then add this image to the pre-reconstructed image pixel by pixel to obtain a high-resolution image and output it.

2. The method for arbitrary-scale super-resolution reconstruction of multiple-frame images according to claim 1, characterized in that: The specific method of aligning the acquired multiple frames of continuous shooting images by using the translation transformation in the affine transformation is as follows: one frame of the multiple frames of continuous shooting images is selected as a reference image, and each affine matrix for the translation transformation of each of the remaining frames of the image is calculated through the enhanced correlation coefficient maximization algorithm ECC based on the reference image, and each of the remaining frames of the image is translated using the corresponding affine matrix to achieve alignment with the reference image.

3. The method for arbitrary-scale super-resolution reconstruction of multiple-frame images according to claim 1, characterized in that: The method of extracting a shallow feature map through a convolution layer is as follows: extracting a shallow feature map of each frame image through a 3*3 convolution layer Wherein, N represents the maximum number of frames of the multi-frame continuous shooting image.

4. The method for arbitrary-scale super-resolution reconstruction of multiple-frame images according to claim 3, characterized in that: The extracted shallow feature map is subjected to feature fusion through a multi-frame feature fusion network to obtain a fused feature map, specifically: S5.1.1: Take a frame in the shallow feature map as the reference feature map, record is the reference feature map; S5.1.2: Multiply all shallow feature maps pixel by pixel by the reference feature map to enlarge the difference between feature maps. The calculation process is as follows: in, represents the enlarged feature map and n=1,…,N; S5.1.3: Calculate the difference feature map R other than the reference frame through the enlarged feature map n , the calculation process is as follows: R n = S5.1.4: Calculate the attention fusion weight map for the current feature map through the difference feature map. The calculation process is as follows: A n =Sigmoid(R n ), n = 2, ..., N, where A n is the attention fusion weight map, R n is the difference feature map; S5.1.5: Use the attention fusion weight map to calculate the feature map for each frame except the reference frame. The calculation process is as follows: n=2,…,N,where F n is the feature map used for fusion, is the initial shallow feature map; S5.1.6: All feature maps F obtained for fusion n , n = 2, ..., N and the initially selected reference feature map Concatenate in the channel dimension and send it to the residual network to obtain the fused feature map.

5. The method for arbitrary-scale super-resolution reconstruction of multiple-frame images according to claim 1, characterized in that: The fused feature map is obtained by passing the fused feature map through a multi-scale feature extraction network. Specifically, the fused feature map is first projected into the deep feature space through a projection convolution layer, and then enters the downsampling stage, and the highest-level features are extracted through the ST layer using the attention mechanism, which is recorded as Then, the features are downsampled through the downsampling layer and then passed through the ST layer to obtain secondary features. Downsample again and get low-level features through the ST layer Then enter the upsampling stage, After upsampling Add pixel by pixel and then pass through the ST layer to obtain the secondary fusion features of the first stage Upsample again with After pixel-by-pixel addition, the ST layer is used to obtain the highest-level fusion features of the first stage. Then Then go through the same downsampling and upsampling process to obtain the low-level features of the second stage Secondary fusion features and the highest level fusion features Finally, the low-level features, secondary fusion features, and highest-level fusion features obtained in the two stages are concatenated in the channel dimension respectively, and three scales of deep fusion features Feat0, Feat1, and Feat2 are obtained through three different output convolution projections; wherein the ST layer is a Swin Transformer layer.

6. The method for arbitrary-scale super-resolution reconstruction of multiple-frame images according to claim 1, characterized in that: The method of mining implicit features in the multi-scale feature map is specifically as follows: according to the set target super-resolution magnification factor, the normalized coordinates of each pixel point of the target high-resolution image are obtained, each pixel coordinate is linearly normalized to the range of [-1,1], and the implicit feature vector of the corresponding coordinate is collected in the multi-scale feature map according to the normalized coordinate.

7. The method for arbitrary-scale super-resolution reconstruction of multiple-frame images according to claim 1, characterized in that: The implicit features are spliced ​​in the channel dimension and then input into the implicit feature reconstruction network to obtain the pre-reconstructed image. Specifically, the spliced ​​implicit feature vector is input into the implicit feature reconstruction network, and a feature vector of each pixel used for reconstruction is obtained through a series of MLP networks composed of a fully connected layer, a Dropout layer, and a Relu activation function layer. All feature vectors are spliced ​​into a feature map according to the original coordinates, and finally the pre-reconstructed image is output through a convolution layer.

Images