Multi-view image super-resolution method based on deep learning and geometry perception

Abstract

The present invention belongs to the technical field of intelligent processing of digital images, and specifically is a deep learning multi-view image super-resolution method based on geometric perception. The method of the present invention comprises: inputting all multi-view images in a multi-view scene and their corresponding depth maps, internal and external parameters of the camera into a reference image synthesis network, and synthesizing multiple reference images of each view image using geometric information; inputting the view image and the synthesized reference image into a reference super-resolution network, and assisting the super-resolution of the view image by using the synthesized reference image. The advantage of the present invention is that it utilizes the information of all multi-view images in the scene for super-resolution. Experimental results show that the present invention extracts useful information from all view images, which can significantly improve the performance of the super-resolution model, thereby improving the visual experience based on multi-view image applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent processing of digital images, and specifically relates to a super-resolution method, and more specifically to a deep learning multi-view image super-resolution method based on geometric perception. Background Art

[0002] Super-resolution is the task of reconstructing a high-resolution image from an observed low-resolution image. Super-resolution has important applications in surveillance equipment, satellite imagery, and medical imaging.

[0003] High-resolution images not only significantly improve the user's visual experience but also save storage space and network bandwidth. In multi-view multimedia applications such as virtual reality, street view, and live sports broadcasts, there's often a trade-off between high-resolution multi-view images and storage space and network bandwidth. Multi-view super-resolution effectively addresses the issue of limited transmission of low-resolution images, making it a valuable research topic.

[0004] Super-resolution has been widely studied, and many super-resolution methods have been proposed in academia and industry. [1] Since the proposal of , the super-resolution method based on neural network has far surpassed the traditional method and has become the mainstream of super-resolution methods.

[0005] Image super-resolution is one of the earliest super-resolution tasks, and some methods [2-4] By designing a sophisticated model structure to learn high-frequency details of images from the dataset, other methods [5] Attempts are made to use self-reference to find repeated texture patterns to assist super-resolution. However, since high-frequency texture information is destroyed in the downsampled low-resolution image, image super-resolution is difficult to recover high-resolution texture details. In order to directly provide useful high-frequency texture details to the network, many reference-based image super-resolution methods have been proposed. [6-7] They try to find small patches that appear repeatedly across scales in the reference image. Since videos have a temporal dimension, video super-resolution has more relevant high-frequency detail information than image super-resolution. They use optical flow [8] or Deformable Convolution [9] Perform explicit or implicit video frame alignment for super-resolution. In addition, some video super-resolution methods

[10] Use recurrent neural networks to obtain high-frequency texture information of distant video frames. Summary of the Invention

[0006] To overcome the shortcomings of the existing technology, the present invention aims to provide a deep learning multi-view image super-resolution method based on geometric perception. By leveraging geometric information, high-frequency texture information from all multi-view images of the same scene is extracted and utilized to improve the quality of super-resolution results.

[0007] The present invention provides a deep learning multi-view image super-resolution method based on geometric perception, which specifically comprises the following steps:

[0008] (1) All multi-view images in the same multi-view scene and their corresponding depth maps, camera internal and external parameters are input into the reference image synthesis network, which uses scene geometry information to align images and depth maps;

[0009] (2) The reference image synthesis network uses the distance geometry information obtained in step (1) to extract high-frequency details in the aligned images to synthesize multiple multi-view reference images and multiple adjacent view reference images for each view image;

[0010] (3) Train a reference super-resolution network, using all the reference images synthesized in step (2) as an aid to super-resolve the low-resolution input view map.

[0011] Furthermore, in step (1), the reference picture synthesis network uses scene geometry information to align the image and depth map according to Figure 2 As shown in the figure, the specific approach is as follows: the reference image synthesis network first upsamples the depth map to the super-resolution target multiple. The super-resolution upsampling target multiple of this method is 4 times, and the internal and external parameters of the camera are modified according to the super-resolution target multiple; then, using this geometric information, all multi-view images and depth maps are aligned to the view image and depth map to be super-resolved through homography transformation. The alignment principle is as follows:

[0012]

[0013]

[0014]

[0015] Its meaning is to align the i-th multi-view image and depth map to the view image and depth map to be super-resolved. The subscripts i, T, and i→T respectively represent the i-th multi-view data, the view data to be super-resolved, and align the i-th multi-view data to the view to be super-resolved. The superscripts lr and hr respectively indicate that the data is low-resolution and high-resolution. D, P, and I respectively represent the multi-view depth map, the multi-view camera internal and external parameters, and the multi-view image. UpSample represents the upsampling operation, and W represents the homography transformation.

[0016] Furthermore, in step (1), the homography transformation is as follows:

[0017]

[0018] in, The subscripts i and T represent the i-th multi-view data and the super-resolution view data, respectively. i and p T is the coordinate of the corresponding pixel in the i-th multi-view data and the super-resolution view data. K, R, and t are the camera intrinsic parameter, camera extrinsic parameter rotation matrix, and camera extrinsic parameter translation matrix, respectively.

[0019] The beneficial effect of step (1) is that the reference image synthesis network explicitly uses scene geometry information to align multi-view images through homography. Current image super-resolution and video super-resolution methods do not use geometric information for alignment, but instead let the network learn alignment implicitly. However, due to the large spacing between viewpoints, the span of multi-view image alignment is very large, which increases the difficulty of implicit alignment and causes errors.

[0020] Furthermore, in step (2), the reference image synthesis network extracts high-frequency details in the aligned images to synthesize multiple multi-view reference images and multiple adjacent view reference images for each view image, according to Figure 3 As shown in the figure, the specific approach is as follows: the reference image synthesis network uses the aligned depth map obtained from the homography transformation as the distance geometry information, and first divides all the aligned depth maps into small blocks, and then assigns the average depth value of each block to this depth block. The purpose of taking the average value is to reduce the impact of inaccurate depth maps. The principle is as follows:

[0021]

[0022]

[0023] Where, 1≤j≤H×W / 16 2 , j represents the jth small square, H and W represent the height and width of the depth map, the side length of each square is 16, unfold and mean represent the depth map block and depth value averaging operations respectively, D i→T,j and Represents the i-th multi-view aligned depth map, the j-th small square of the aligned depth map, and the j-th average small square of the aligned depth map; the present invention compares the depth values ​​of the depth blocks at each position to obtain the index of the minimum depth block; then, based on the obtained index, the image blocks are extracted from the aligned view image obtained from the homography transformation; finally, the extracted image blocks are combined into a synthetic reference image, the principle of which is as follows:

[0024]

[0025]

[0026] R=fold(R1,R2,…,R J ), (9)

[0027] Where J = H × W / 16 2 , H j 、R j and R represent the index of the j-th depth block, the j-th block image obtained from the index, and the synthetic reference image composed of all the block images, respectively. Fold represents the operation of combining into a reference image. The reference image synthesis network repeats this process 6 times to obtain 6 synthetic multi-view reference images obtained by the first 6 small depth map indexes. Here, the superscript M indicates that the reference composite image is synthesized from all views.

[0028] Furthermore, in step (2), the present invention inputs the aligned depth map and aligned view map corresponding to the 6 adjacent view maps of each view into the reference picture synthesis network to obtain 6 synthetic adjacent view reference maps. Here, the superscript N indicates that the reference composite image is synthesized from adjacent views.

[0029] The beneficial effect of step (2) is that the reference picture synthesis network extracts the high-frequency information required for super-resolution from all aligned pictures by using distance geometry information and combines it into the synthetic reference picture. Figure 5 As shown, the reference image synthesized by this invention has abundant texture details, such as the clearer English letters on the truck. Current image super-resolution methods can only utilize information from the original image, and video super-resolution methods can only utilize information from adjacent frames. However, the reference image synthesis network of this invention utilizes information from all viewpoints in a multi-view scene, a feature not available in previous super-resolution networks.

[0030] Furthermore, in step (3), according to Figure 4 As shown in the figure, the reference super-resolution network first performs feature encoding and uses a reference image feature extractor to extract multi-scale reference features of the multi-view reference image and the adjacent view reference image, and then fuses them into the input view features in a residual manner at each scale; the reference super-resolution network uses an input view feature extractor to extract the input view features; the entire reference super-resolution network consists of convolutional layers and ReLU, and the upsampling method of the input view features at each scale is Pixelshuffle.

[0031] Furthermore, in step (3), the loss function used by the reference super-resolution network is the reconstruction loss function, and the output of the reference super-resolution network is The target output is Therefore, the loss function is calculated as follows:

[0032]

[0033] The beneficial effect of step (3) is that the reference super-resolution network extracts and transfers the high-frequency information in the synthesized reference image to the super-resolution process in a multi-scale residual manner through the reference image feature extractor, thereby improving the quality of super-resolution. As shown in Table 1, other compared solutions include EDSR [2] 、RCAN [4] , RBPN [8] 、IconVSR

[10] TTSR [6] and MASA [7] ,From the last row of the figure, we can see that our indicators perform best compared with other solutions. Figure 5 As shown in the result graph of the present invention, our reference super-resolution network makes full use of the synthetic reference image to generate high-quality high-resolution perspective images with rich details, such as the English letters on the truck are clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a multi-view super-resolution network framework diagram of the present invention.

[0035] Figure 2 This is a structural diagram of the reference picture synthesis network of the present invention using scene geometry information.

[0036] Figure 3 This is a structural diagram of the reference picture synthesis network of the present invention using distance geometry information.

[0037] Figure 4 This is a structural diagram of the reference super-resolution network of the present invention.

[0038] Figure 5 Comparison of the super-resolution visualization results of the multi-view super-resolution network of the present invention and other solutions. DETAILED DESCRIPTION

[0039] The embodiments of the present invention are described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0040] use Figure 1The model framework in

[15] uses a reference image synthesis network to synthesize multiple reference images for each input view, and then a reference super-resolution network uses the synthesized reference images to perform super-resolution model training on the input view map.

[0041] The specific steps are:

[0042] (1) Before training, the present invention uses the Tanks and Temple dataset to train the reference super-resolution network. In the dataset, except for the Train, M60, Playground, and Truck scenes used for testing, the other scenes are used for training.

[0043] All multi-view images of the same scene and their corresponding depth maps and camera parameters are input into the reference image synthesis network. Figure 2 As shown in , the reference image synthesis network uses the scene geometry information of the input image, depth map and camera parameters to align all view images and their depth maps to each view image and depth map in the scene through homography transformation. Figure 3 As shown, the reference image synthesis network uses the aligned depth maps as distance geometry information, extracts high-resolution image blocks from the aligned view maps, and then combines these blocks into a reference image. The synthesized reference image is not only aligned with the input view to be super-resolved but also contains rich high-frequency texture details, significantly improving the quality of super-resolution.

[0044] (2) During training, randomly crop the image to 80×160 and normalize the image pixels to the range of -1 to 1. The initial learning rate is set to 10 -4 , the final learning rate is 10 -7 , and adopt the cosine annealing learning rate descent method, the batch size is set to 2, and the training is 200,000 iterations. The optimizer used is Adam to train the reference super-resolution network. Figure 4 As shown, the reference super-resolution network uses a reference image feature extractor to extract multi-scale reference features from the multi-view reference image and the adjacent view reference image, and then fuses them into the input view features in a residual manner at each scale. The present invention uses a reconstruction loss on the high-resolution images generated by the reference super-resolution network to constrain the training of the entire network.

[0045] (3) During testing, only the trained model parameters need to be loaded without cropping the image. By directly inputting each input perspective and its synthesized reference image into the reference super-resolution network, a high-resolution reference image can be obtained.

[0046] The multi-view super-resolution network in this invention is superior to other schemes in PSNR, SSIM, LPIPS

[12] The comparison of indicators is shown in Table 1. The higher the PSNR and SSIM indicators are, the better the image quality is, and the smaller the LPIPS indicator is, the better the image is perceived by humans. [2] 、RCAN [4] , RBPN [8] 、IconVSR

[10] TTSR [6] and MASA [7] ,From the last row of the figure, we can see that our indicator performs the best ,compared to other solutions.

[0047] Figure 5 This figure compares the super-resolution visualization results of our multi-view super-resolution network with those of other approaches. As can be seen, the synthesized reference image of our invention possesses abundant texture details, such as the sharper English letters on the truck, demonstrating that our invention extracts useful information from images of all viewpoints. The resulting graph demonstrates that our reference super-resolution network fully utilizes the synthesized reference image to generate high-quality, high-resolution images with rich detail.

[0048] Table 1

[0049]

[0050] References

[0051] [1]Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.2016.ImageSuper-Resolution Using Deep Convolutional Networks.IEEE Transactions onPatternAnalysis and Machine Intelligence 38,2(2016),295–307.

[0052] [2]Bee Lim,Sanghyun Son,Heewon Kim,Seungjun Nah,and Kyoung MuLee.2017.Enhanced Deep Residual Networks for Single Image Super-Resolution.InTheIEEE Conference on Computer Vision and Pattern Recognition(CVPR)Workshops.

[0053] [3] Ben Niu, Weilei Wen, Wenqi Ren, Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang, Xiaochun Cao, and Haifeng Shen. 2020. Single Image Super-Resolution via a Holistic Attention Network. In Computer Vision–ECCV 2020, 191–207.

[0054] [4]Yulun Zhang,Kunpeng Li,Kai Li,Lichen Wang,Bineng Zhong,and YunFu.2018.Image Super-Resolution Using Very Deep Residual Channel Attention Networks.In ECCV.

[0055] [5]Shangchen Zhou,Jiawei Zhang,Wangmeng Zuo,and Chen ChangeLoy.2020.Cross-Scale Internal Graph Neural Network for Image Super-Resolution.In Advances in Neural Information Processing Systems,Vol.33.CurranAssociates,Inc.,3499–3509.

[0056] [6] Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and BainingGuo. 2020. Learning Texture Transformer Network for Image Super-Resolution. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[0057] [7]Liying Lu,Wenbo Li,Xin Tao,Jiangbo Lu,and Jiaya Jia.2021.MASA-SR:MatchingAcceleration and Spatial Adaptation for Reference-Based Image Super-Resolution.In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition(CVPR).6368–6377.

[0058] [8]Kelvin C.K.Chan,Xintao Wang,Ke Yu,Chao Dong,and Chen ChangeLoy.2021.BasicVSR:The Search for Essential Components in Video Super-Resolution and Beyond.In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition(CVPR).4947–4956.

[0059] [9]Xintao Wang,Kelvin C.K.Chan,Ke Yu,Chao Dong,and Chen ChangeLoy.2019.EDVR:Video Restoration With Enhanced Deformable ConvolutionalNetworks.In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition(CVPR)Workshops.

[0060]

[10] Muhammad Haris,Gregory Shakhnarovich,and NorimichiUkita.2019.Recurrent Back-Projection Network for Video Super-Resolution.InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition(CVPR).

[0061]

[11] Wenzhe Shi,Jose Caballero,Ferenc Huszar,Johannes Totz,AndrewP.Aitken,Rob Bishop,Daniel Rueckert,and Zehan Wang.2016.Real-Time SingleImage and Video Super-Resolution Using an Efficient Sub-Pixel ConvolutionalNeural Network.In Proceedings of the IEEE Conference on Computer Vision andPattern Recognition(CVPR).

[0062]

[12] Richard Zhang,Phillip Isola,Alexei A.Efros,Eli Shechtman,andOliverWang.2018.The Unreasonable Effectiveness of Deep Features as aPerceptual Metric.In Proceedings of the IEEE Conference on Computer VisionandPattern Recognition(CVPR).。

Claims

1. A deep learning multi-view image super-resolution method based on geometric perception, characterized by: The specific steps are: (1) All multi-view images in the same multi-view scene and their corresponding depth maps, camera internal and external parameters are input into the reference image synthesis network, which uses scene geometry information to align images and depth maps; (2) The reference image synthesis network uses the distance geometry information obtained in step (1) to extract high-frequency details in the aligned images to synthesize multiple multi-view reference images and multiple adjacent view reference images for each view image; (3) Train the reference super-resolution network, using all the reference images synthesized in step (2) as auxiliary to perform super-resolution training on the low-resolution input view map; In step (1), the reference picture synthesis network uses scene geometry information to align the image and depth map. Specifically, the reference picture synthesis network first upsamples the depth map to the super-resolution target multiple. The super-resolution upsampling target multiple of this method is 4 times, and the internal and external parameters of the camera are modified according to the super-resolution target multiple. Then, these geometric information are used to align all multi-view images and depth maps to the view images and depth maps to be super-resolved through homography transformation. The alignment principle is as follows: Its meaning is to align the i-th multi-view image and depth map to the view image and depth map to be super-resolved; where the subscripts i, T and i→T respectively represent the i-th multi-view data, the view data to be super-resolved and the alignment of the i-th multi-view data to the view to be super-resolved, the superscripts lr and hr respectively represent the data of low resolution and high resolution, D, P and I respectively represent the multi-view depth map, the multi-view camera internal and external parameters and the multi-view image, UpSample represents the upsampling operation, and W represents the homography transformation; In step (2), the high-frequency details in the aligned images are extracted to synthesize multiple multi-view reference images and multiple adjacent view reference images for each view image. The specific method is as follows: the reference image synthesis network uses the aligned depth map obtained from the homography transformation as the distance geometry information, and first divides all the aligned depth maps into small blocks, and then assigns the average depth value of each block to this depth block. The principle is as follows: Where, 1≤j≤H×W / 16 2 , j represents the jth small square, H and W represent the height and width of the depth map, the side length of each square is 16, unfold and mean represent the depth map block and depth value averaging operations respectively, D i→T,j and Represents the i-th multi-view aligned depth map, the j-th small square of the aligned depth map, and the j-th average small square of the aligned depth map; compares the depth values ​​of the depth blocks at each position to obtain the index of the minimum depth block; then extracts image blocks from the aligned view image obtained from the homography transformation based on the obtained index; finally, the extracted image blocks are combined into a synthetic reference image, the principle of which is as follows: R=fold(R1,R2,…,R J ), (9) Where J = H × W / 16 2 , H j 、R j and R represent the index of the j-th depth block, the j-th small block image obtained from the index, and the synthetic reference image composed of all small block images, respectively. Fold represents the operation of combining into a synthetic reference image. The reference image synthesis network repeats this process 6 times to obtain 6 synthetic multi-view reference images obtained by the first 6 small depth map indexes. Here, the superscript M indicates that the reference composite image is synthesized from all views.

2. The method according to claim 1, characterized in that The formula for the homography transformation is as follows: in, t r =t i -R r t T , the subscripts i and T represent the i-th multi-view data and the super-resolution view data, respectively, i and p T are the coordinates of the corresponding pixels in the i-th multi-view data and the super-resolution view data; K, R and t are the camera intrinsic parameter, camera extrinsic parameter rotation matrix, and camera extrinsic parameter translation matrix respectively.

3. The method according to claim 1, characterized in that The specific process of step (3) is as follows: the reference super-resolution network first performs feature encoding, and uses a reference image feature extractor to extract multi-scale reference features of the multi-view reference image and the adjacent view reference image, and then fuses them into the features of the input view in a residual manner at each scale; The reference super-resolution network uses an input view feature extractor to extract input view features; The entire reference super-resolution network consists of convolutional layers and ReLU layers, and the upsampling method of the input view features at each scale is Pixelshuffle.

4. The method according to claim 1 or 3, characterized in that The loss function used by the reference super-resolution network is the reconstruction loss function, and the output of the reference super-resolution network is The target output is The loss function is calculated as follows:

Images