A spatial super-resolution reconstruction method for light field images
By constructing a convolutional neural network combined with a light field camera and a 2D camera, the problems of texture restoration and parallax structure preservation of light field images at large reconstruction scales were solved, and high-quality spatial super-resolution reconstruction of light field images was achieved.
Patent Information
- Application Number
- CN202111405987.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-24
AI Technical Summary
Existing light field image spatial super-resolution reconstruction methods are difficult to effectively recover high-frequency texture information at large reconstruction scales, and are prone to visual artifacts and damage to parallax structures.
A convolutional neural network is constructed, which combines a light field camera and a traditional 2D camera. Multi-scale features are extracted through an encoder, feature registration is performed by an aperture-level feature registration module, feature fusion is performed by a light field feature enhancement module, and a high spatial resolution light field image is reconstructed using a decoder. At the same time, a pyramid network is used for multi-scale reconstruction.
It effectively recovers the high-frequency texture information of the light field image, avoids visual artifacts, and preserves the parallax structure, thus achieving high-quality spatial super-resolution reconstruction of the light field image.
Smart Images

Figure CN114359041B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image super-resolution reconstruction technology, and in particular to a light field image space super-resolution reconstruction method. Background Art
[0002] Unlike conventional digital cameras, light field cameras can collect the intensity (i.e., spatial information) and direction information (i.e., angle information) of light in a scene, thereby recording the real world more realistically. At the same time, the rich information contained in the 4-dimensional (4D) light field images collected by light field cameras promotes many applications, such as refocusing, depth estimation, virtual / augmented reality, etc. Current commercial-grade light field cameras use microlens arrays to separate light rays from different directions passing through the same location in the scene, thereby simultaneously recording spatial information and angle information on the sensor plane. However, due to the limited resolution of sensors that share the spatial and angle dimensions, the spatial resolution of the collected 4D light field images is inevitably reduced while providing high angle sampling (or high angle resolution). Therefore, improving the spatial resolution of 4D light field images has become an important issue that needs to be solved in the field of light field research.
[0003] Generally speaking, 4D light field images include a variety of visualization methods that can be converted to each other, such as a sub-aperture image (SAI) array based on 2-dimensional (2D) spatial information display, a micro-lens image (MLI) array based on 2D angle information display, and an epipolar plane image (EPI) that combines 1D spatial information with 1D angle information. Intuitively, improving the spatial resolution of 4D light field images is to improve the resolution of each 2D SAI in the 4D light field image. Therefore, a direct approach is to apply existing 2D image super-resolution reconstruction methods, such as the deep back-projection network proposed by Haris et al. and the deep Laplacian pyramid network proposed by Lai et al., to each SAI independently, but this approach ignores the information embedded in the angular domain of the 4D light field image, and it is difficult to ensure the angular consistency of the super-resolution results. Therefore, exploring the high-dimensional structural characteristics of 4D light field images is the key to designing a spatial super-resolution reconstruction method for 4D light field images. At present, the spatial super-resolution reconstruction methods for 4D light field images can be roughly divided into two categories: optimization-based and learning-based.
[0004] Optimization-based methods usually use estimated disparity or depth information to model the relationship between the SAIs of 4D light field images, and then express the spatial super-resolution reconstruction of 4D light field images as an optimization problem. However, the disparity or depth information inferred from low spatial resolution light field images is not very reliable, so the performance of optimization-based methods is quite limited.
[0005] Learning-based methods explore the intrinsic high-dimensional structure of 4D light field images in a data-driven manner and use this to learn the nonlinear mapping between low spatial resolution light field images and high spatial resolution light field images. For example, Yeung et al. used spatial-angular separable convolution to iteratively utilize the spatial and angular information of 4D light field images. Wang et al. developed a spatial-angular interaction network to fuse the spatial and angular information of 4D light field images. Jin et al. proposed a novel fusion mechanism to utilize the compensation information between SAIs and restore the parallax details of 4D light field images through a two-stage network. Although the above methods have achieved good performance at low reconstruction scales (such as 2×), they still cannot effectively restore sufficient texture and detail information at large reconstruction scales (such as 8×). This is because low-resolution light field images contain limited spatial and angular information, and the above methods can only infer the details lost due to low resolution through the information inside the 4D light field image. Boominathan et al. proposed a spatial super-resolution reconstruction method for light field images using mixed input, which improves the spatial resolution of 4D light field images by introducing an additional high-resolution 2D image as supplementary information. However, the average fusion mechanism in this method easily blurs the reconstruction results, and independently processing each SAI will destroy the parallax structure of the reconstructed light field image.
[0006] In summary, although current related research has achieved good spatial super-resolution reconstruction effects of light field images at low reconstruction scales, there are still some shortcomings in dealing with large reconstruction scales (such as 8×). In particular, there is still room for improvement in restoring the high-frequency texture information of reconstructed light field images, avoiding visual artifacts, and preserving the parallax structure. Summary of the invention
[0007] The technical problem to be solved by the present invention is to provide a light field image spatial super-resolution reconstruction method, which combines a light field camera with a traditional 2D camera to form a heterogeneous imaging system. The light field camera provides rich angle information and limited spatial information, while the traditional 2D camera only collects the intensity information of light to obtain sufficient spatial information. The angle information and spatial information obtained by both can be fully utilized to reconstruct a high spatial resolution light field image with high quality, and restore the texture and detail information of the reconstructed light field image, while avoiding ghost artifacts caused by parallax and retaining the parallax structure.
[0008] The technical solution adopted by the present invention to solve the above technical problem is: a light field image spatial super-resolution reconstruction method, characterized by comprising the following steps:
[0009] Step 1: select Num color three-channel low spatial resolution light field images with a spatial resolution of W×H and an angular resolution of V×U, the corresponding Num color three-channel 2D high-resolution images with a resolution of αW×αH, and the corresponding Num color three-channel reference high spatial resolution light field images with a spatial resolution of αW×αH and an angular resolution of V×U; wherein Num>1, α represents the spatial resolution improvement multiple, and the value of α is greater than 1;
[0010] Step 2: Build a convolutional neural network as a spatial super-resolution network: The spatial super-resolution network includes an encoder for extracting multi-scale features, an aperture-level feature registration module for registering light field features and 2D high-resolution features, a shallow feature extraction layer for extracting shallow features from low spatial resolution light field images, a light field feature enhancement module for fusing light field features and 2D high-resolution features, a spatial attention block for alleviating registration errors in coarse-scale features, and a decoder for reconstructing latent features into a light field image;
[0011] For the encoder, it consists of a first convolutional layer, a second convolutional layer, a first residual block and a second residual block connected in sequence. The input end of the first convolutional layer receives three inputs in parallel, which are a single-channel image L of a low spatial resolution light field image with a spatial resolution of W×H and an angular resolution of V×U. LR The width of the reconstructed image after spatial resolution upsampling is α s W×V and height is α s The H×U sub-aperture image array is denoted as A width of α s W and height is α s The single-channel image of the blurred 2D high-resolution image of H is denoted as and a width of α s W and height is α s A single-channel image of the 2D high-resolution image of H, denoted as I HR , the output of the first convolutional layer is for Output 64 images with a width of α s W×V and height is α s The H×U feature map will be targeted The set of all output feature maps is recorded as The output of the first convolutional layer is Output 64 images with a width of α s W and height is αs The feature map of H will be The set of all output feature maps is recorded as The output of the first convolutional layer is I HR Output 64 images with a width of α s W and height is α s The feature map of H will be HR The set of all output feature maps is recorded as Y HR,0 ; The input end of the second convolutional layer receives three inputs in parallel, namely All feature maps in All feature maps and Y HR,0 All feature maps in the second convolutional layer are Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second convolutional layer is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second convolutional layer is for Y HR,0 Output 64 width And the height is The feature map of Y HR ,0 The set of all feature maps output is recorded as Y HR,1 ; The input end of the first residual block receives three inputs in parallel, namely All feature maps in All feature maps and Y HR,1 For all feature maps in , the output of the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the first residual block is for Y HR,1 Output 64 width And the height is The feature map of Y HR,1The set of all output feature maps is recorded as Y HR,2 ; The input end of the second residual block receives three inputs in parallel, namely All feature maps in All feature maps and Y HR,2 For all feature maps in , the output of the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second residual block is for Y HR,2 Output 64 width And the height is The feature map of Y HR,2 The set of all output feature maps is recorded as Y HR,3 ;in, is a single-channel image L obtained by transforming a low spatial resolution light field image with a spatial resolution of W×H and an angular resolution of V×U LR The width of the reconstructed image after bicubic interpolation upsampling is α s W×V and height is α s H×U sub-aperture image array, To pass I HR First perform bicubic interpolation downsampling and then perform bicubic interpolation upsampling to obtain, α s represents the spatial resolution sampling factor, α s 3 =α, the upsampling factor of bicubic interpolation upsampling and the downsampling factor of bicubic interpolation downsampling are both α s The size of the convolution kernel of the first convolution layer is 3×3, the convolution step is 1, the number of input channels is 1, and the number of output channels is 64. The size of the convolution kernel of the second convolution layer is 3×3, the convolution step is 2, the number of input channels is 64, and the number of output channels is 64. The activation function used in the first and second convolution layers is "ReLU";
[0012] For the aperture-level feature registration module, its input receives three types of feature maps. The first type is All feature maps in the second category are All feature maps in the third category include four inputs, namely Y HR,0 All feature maps in YHR,1 All feature maps in Y HR,2 All feature maps in Y HR,3 All feature maps in the aperture level feature registration module are first All feature maps in Y HR,0 All feature maps in Y HR,1 All feature maps in Y HR,2 All feature maps and Y HR,3 All feature maps in are replicated V×U times, so that All feature maps in Y HR,1 All feature maps in Y HR,2 All feature maps and Y HR,3 The width of all feature maps in becomes And the height becomes Even if the size and and make Y HR,0 The width of all feature maps in becomes α s W×V and the height becomes α s H×U, that is, the size Match the size of the feature map in ; then All feature maps and All feature maps in are block matched, and after the block matching is completed, a width of And the height is The coordinate index map of CI ; Then according to P CI , Y HR,1 All feature maps in All feature maps in are spatially aligned to obtain 64 feature maps with a width of And the height is The set of all the obtained registration feature maps is recorded as F Align,1 Similarly, according to P CI , Y HR,2 All feature maps in All feature maps in are spatially aligned to obtain 64 feature maps with a width of And the height is The set of all the obtained registration feature maps is recorded as F Align,2 According to P CI , Y HR,3 All feature maps in All feature maps in are spatially aligned to obtain 64 feature maps with a width of And the height is The set of all the obtained registration feature maps is recorded as FAlign,3 ; Then for P CI Perform bicubic interpolation upsampling to obtain a width of α s W×V and height is α s The coordinate index map of H×U is denoted as Finally, according to Y HR,0 All feature maps in All feature maps in the spatial position are registered to obtain 64 images with a width of α s W×V and height is α s The set of all the obtained registration feature maps is denoted as F Align,0 ; The output of the aperture level feature registration module outputs F Align,0 All feature maps in F Align,1 All feature maps in F Align,2 All feature maps and F Align,3 All feature maps in ; the accuracy measure for block matching is the texture and structure similarity index, the size of the block used for block matching is 3×3, and the upsampling factor of bicubic interpolation upsampling is α s ;
[0013] For the shallow feature extraction layer, it consists of a fifth convolutional layer, the input of which receives a single-channel image L of a low spatial resolution light field image with a spatial resolution of W×H and an angular resolution of V×U LR The reorganized sub-aperture image array is of width W×V and height H×U. The output of the fifth convolutional layer outputs 64 feature maps of width W×V and height H×U. The set of all the output feature maps is denoted as F LR ; Among them, the size of the convolution kernel of the fifth convolution layer is 3×3, the convolution step is 1, the number of input channels is 1, the number of output channels is 64, and the activation function used in the fifth convolution layer is "ReLU";
[0014] For the light field feature enhancement module, it is composed of a first enhanced residual block, a second enhanced residual block and a third enhanced residual block connected in sequence, and the input end of the first enhanced residual block receives F Align,1 All feature maps and F LR All feature maps in the first enhanced residual block have a width of 64. And the height is The set of all the output feature maps is recorded as F En,1 ; The input end of the second enhanced residual block receives F Align,2 All feature maps and F En,1 All feature maps in the second enhanced residual block output 64 images with a width of And the height is The set of all the output feature maps is recorded as F En,2 ; The input end of the third enhanced residual block receives F Align,3 All feature maps and F En,2 All feature maps in the third enhanced residual block output 64 pixels with a width of And the height is The set of all the output feature maps is recorded as F En,3 ;
[0015] For the spatial attention block, it consists of the sixth convolutional layer and the seventh convolutional layer connected in sequence. The input end of the sixth convolutional layer receives F Align,0 All feature maps in the , the output of the sixth convolutional layer outputs 64 images with a width of α s W×V and height is α s H×U spatial attention feature map, the set of all output spatial attention feature maps is recorded as F SA1 ; The input of the seventh convolutional layer receives F SA1 All spatial attention feature maps in the , the output of the seventh convolutional layer outputs 64 images with a width of α s W×V and height is α s H×U spatial attention feature map, the set of all output spatial attention feature maps is recorded as F SA2 ; F Align,0 All feature maps in F SA2 All spatial attention feature maps in are multiplied element by element, and the set of all feature maps obtained is recorded as F WA,0 ; F WA,0 All feature maps in are used as all feature maps outputted by the output of the spatial attention block; among them, the convolution kernel size of the sixth convolution layer and the seventh convolution layer are both 3×3, the convolution step size is 1, the number of input channels is 64, the number of output channels is 64, the activation function used in the sixth convolution layer is "ReLU", and the activation function used in the seventh convolution layer is "Sigmoid";
[0016] For the decoder, it consists of the third residual block, the fourth residual block, the sub-pixel convolution layer, the eighth convolution layer and the ninth convolution layer connected in sequence. The input end of the third residual block receives F En,3 All feature maps in the third residual block output 64 images with a width of And the height is The set of all the output feature maps is recorded as F Dec,1 ; The input end of the fourth residual block receives F Dec,1 All feature maps in , the output of the fourth residual block outputs 64 images with a width of And the height is The set of all the output feature maps is recorded as F Dec,2 ; The input of the sub-pixel convolution layer receives F Dec,2 For all feature maps in , the output of the sub-pixel convolutional layer outputs 256 images with a width of And the height is The feature map is 256 images with a width of And the height is The feature map is further converted into 64 images with a width of α s W×V and height is α s The feature map of H×U, the set of all the converted feature maps is recorded as F Dec,3 ; The input of the eighth convolutional layer receives F Dec,3 All feature maps in F WA,0 The result of element-by-element addition of all feature maps in the eighth convolutional layer is 64 images with a width of α. s W×V and height is α s The feature map of H×U, the set of all the output feature maps is recorded as F Dec,4 ; The input of the ninth convolutional layer receives F Dec,4 All feature maps in the , the output of the ninth convolutional layer outputs a width of α s W×V and height is α s H×U reconstructed single-channel light field image, and the width of the image is α s W×V and height is α s The reconstructed single-channel light field image of H×U is reorganized into a spatial resolution of α s W×α s H and an angular resolution of V × U high spatial resolution single-channel light field image, denoted as L SR ; Among them, the size of the convolution kernel of the sub-pixel convolution layer is 3×3, the convolution step is 1, the number of input channels is 64, and the number of output channels is 256. The size of the convolution kernel of the eighth convolution layer is 3×3, the convolution step is 1, the number of input channels is 64, and the number of output channels is 64. The size of the convolution kernel of the ninth convolution layer is 1×1, the convolution step is 1, the number of input channels is 64, and the number of output channels is 1. The activation function used by the sub-pixel convolution layer and the eighth convolution layer is "ReLU", and the ninth convolution layer does not use an activation function;
[0017] Step 3: Perform color space conversion on each low spatial resolution light field image, the corresponding 2D high resolution image, and the corresponding reference high spatial resolution light field image in the training set, that is, convert from RGB color space to YCbCr color space, and extract the Y channel image; then reorganize the Y channel image of each low spatial resolution light field image into a sub-aperture image array with a width of W×V and a height of H×U to represent it; then, the sub-aperture image array reorganized from the Y channel images of all low spatial resolution light field images in the training set, the Y channel image of the corresponding 2D high resolution image, and the Y channel image of the corresponding reference high spatial resolution light field image constitute the training set; then construct a pyramid network and use the training set for training. The specific process is as follows:
[0018] Step 3_1: The constructed spatial super-resolution network is replicated three times and cascaded. The weights of each spatial super-resolution network are shared, that is, the parameters are all the same. The overall network composed of three spatial super-resolution networks is defined as a pyramid network; at each pyramid level, the reconstruction scale of the spatial super-resolution network is set to be the same as α s The values are the same;
[0019] Step 3_2: perform two spatial resolution downsampling on the Y channel image of each reference high spatial resolution light field image in the training set, and use the image obtained after downsampling as the label image; perform two spatial resolution downsampling on the Y channel image of each 2D high resolution image in the training set, and use the image obtained after downsampling as the 2D high resolution Y channel image for the first spatial super resolution network in the pyramid network; then, the sub-aperture image array reconstructed from the Y channel images of all low spatial resolution light field images in the training set, the sub-aperture image array reconstructed from the Y channel images of all low spatial resolution light field images in the training set after one spatial resolution upsampling, all the 2D high resolution Y channel images for the first spatial super resolution network in the pyramid network, and all the blurred 2D high resolution Y channel images for the first spatial super resolution network in the pyramid network after one spatial resolution downsampling and one spatial resolution upsampling are input into the first spatial super resolution network in the constructed pyramid network for training, and the α corresponding to the Y channel image of each low spatial resolution light field image in the training set is obtained. s The high spatial resolution Y channel light field image is reconstructed by times; the spatial resolution upsampling and spatial resolution downsampling methods are both bicubic interpolation, and the scales of spatial resolution upsampling and spatial resolution downsampling are the same as α s The values are the same;
[0020] Step 3_3: Perform a single spatial resolution downsampling on the Y channel image of each reference high spatial resolution light field image in the training set, and use the downsampled image as the label image; perform a single spatial resolution downsampling on the Y channel image of each 2D high resolution image in the training set, and use the downsampled image as the 2D high resolution Y channel image for the second spatial super resolution network in the pyramid network; then perform the α corresponding to the Y channel image of all low spatial resolution light field images in the training set on the Y channel image of the training set. s The sub-aperture image array reconstructed from the high spatial resolution Y channel light field image and the α corresponding to the Y channel image of all low spatial resolution light field images in the training set are reconstructed. s The sub-aperture image array of the reconstructed high spatial resolution Y channel light field image obtained after one spatial resolution upsampling, all 2D high-resolution Y channel images for the second spatial super-resolution network in the pyramid network, and all 2D high-resolution Y channel images for the second spatial super-resolution network in the pyramid network after one spatial resolution downsampling and one spatial resolution upsampling are input into the second spatial super-resolution network in the constructed pyramid network for training, and the α corresponding to the Y channel image of each low spatial resolution light field image in the training set is obtained. s 2 The high spatial resolution Y channel light field image is reconstructed by times; the spatial resolution upsampling and spatial resolution downsampling methods are both bicubic interpolation, and the scales of spatial resolution upsampling and spatial resolution downsampling are the same as α s The values are the same;
[0021] Step 3_4: Take the Y channel image of each reference high spatial resolution light field image in the training set as the label image; take the Y channel image of each 2D high resolution image in the training set as the 2D high resolution Y channel image for the third spatial super resolution network in the pyramid network; then take the α corresponding to the Y channel image of all low spatial resolution light field images in the training set as the label image. s 2 The sub-aperture image array reconstructed from the high spatial resolution Y channel light field image and the α corresponding to the Y channel image of all low spatial resolution light field images in the training set are reconstructed. s 2The sub-aperture image array of the reconstructed high spatial resolution Y channel light field image obtained after one spatial resolution upsampling, all 2D high-resolution Y channel images for the third spatial super-resolution network in the pyramid network, and all 2D high-resolution Y channel images for the third spatial super-resolution network in the pyramid network after one spatial resolution downsampling and one spatial resolution upsampling are input into the third spatial super-resolution network in the constructed pyramid network for training, and the α corresponding to the Y channel image of each low spatial resolution light field image in the training set is obtained. s 3 The high spatial resolution Y channel light field image is reconstructed by times; the spatial resolution upsampling and spatial resolution downsampling methods are both bicubic interpolation, and the scales of spatial resolution upsampling and spatial resolution downsampling are the same as α s The values are the same;
[0022] After the training is completed, the optimal weight parameters of all convolution kernels in each spatial super-resolution network in the pyramid network are obtained, that is, a well-trained spatial super-resolution network model is obtained;
[0023] Step 4: arbitrarily select a color three-channel low spatial resolution light field image and a corresponding color three-channel 2D high resolution image as test images; then convert the color three-channel low spatial resolution light field image and the corresponding color three-channel 2D high resolution image from RGB color space to YCbCr color space, and extract the Y channel image; then reorganize the Y channel image of the low spatial resolution light field image into a sub-aperture image array for representation; then reorganize the sub-aperture image array of the reorganized Y channel image of the low spatial resolution light field image, the sub-aperture image array of the reorganized image after a spatial resolution upsampling of the Y channel image of the low spatial resolution light field image, the Y channel image of the 2D high resolution image, and the Y channel image of the 2D high resolution image after a spatial resolution downsampling and a spatial resolution upsampling to obtain a blurred 2D high resolution Y channel The channel image is input into the spatial super-resolution network model, and the test obtains the reconstructed high spatial resolution Y channel light field image corresponding to the Y channel image of the low spatial resolution light field image; then the Cb channel image and Cr channel image of the low spatial resolution light field image are up-sampled by bicubic interpolation respectively to obtain the reconstructed high spatial resolution Cb channel light field image corresponding to the Cb channel image of the low spatial resolution light field image and the reconstructed high spatial resolution Cr channel light field image corresponding to the Cr channel image of the low spatial resolution light field image; finally, the reconstructed high spatial resolution Y channel light field image, the reconstructed high spatial resolution Cb channel light field image and the reconstructed high spatial resolution Cr channel light field image are cascaded in the color channel dimension, and the cascade result is reconverted to the RGB color space to obtain the reconstructed high spatial resolution light field image of the three color channels corresponding to the low spatial resolution light field image.
[0024] In the step 2, the first residual block, the second residual block, the third residual block and the fourth residual block have the same structure, which are composed of a third convolutional layer and a fourth convolutional layer connected in sequence, and the input end of the third convolutional layer in the first residual block receives three inputs in parallel, which are All feature maps in All feature maps and Y HR,1 For all feature maps in , the output of the third convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the first residual block is for Y HR,1 Output 64 width And the height is The feature map of Y HR,1 The set of all output feature maps is recorded as The input end of the fourth convolutional layer in the first residual block receives three inputs in parallel, which are All feature maps in All feature maps and For all feature maps in , the output of the fourth convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the first residual block for All the feature maps outputted, the set of these feature maps is Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the first residual block for All the feature maps outputted, the set of these feature maps is Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the first residual block for Y HR,1 All the feature maps output, the set of these feature maps is Y HR,2 ;
[0025] The input end of the third convolutional layer in the second residual block receives three inputs in parallel, which are All feature maps in All feature maps and Y HR,2 For all feature maps in , the output of the third convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the second residual block is for Y HR,2 Output 64 width And the height is The feature map of Y HR,2 The set of all output feature maps is recorded as The input end of the fourth convolutional layer in the second residual block receives three inputs in parallel, which are All feature maps in All feature maps and For all feature maps in , the output of the fourth convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the second residual block for All the feature maps outputted, the set of these feature maps is Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the second residual block for All the feature maps outputted, the set of these feature maps is Y HR,2 All feature maps in All feature maps in Y are added element by element, and all the obtained feature maps are used as the output of the second residual block for Y HR,2 All the feature maps output, the set of these feature maps is Y HR,3 ;
[0026] The input of the third convolutional layer in the third residual block receives F En,3 All feature maps in the third residual block have a width of 64. And the height is The feature map of the output is recorded as The input of the fourth convolutional layer in the third residual block receives All feature maps in the third residual block have a width of 64. And the height is The feature map of the output is recorded as F En,3 All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the third residual block. The set of these feature maps is F Dec,1 ;
[0027] The input of the third convolutional layer in the fourth residual block receives F Dec,1 All feature maps in the fourth residual block have a width of 64. And the height is The feature map of the output is recorded as The input end of the fourth convolutional layer in the fourth residual block receives All feature maps in the fourth residual block have a width of 64. And the height is The feature map of the output is recorded as F Dec,1 All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the fourth residual block. The set of these feature maps is F Dec,2 ;
[0028] As mentioned above, the size of the convolution kernels of the third convolution layer and the fourth convolution layer in the first residual block, the second residual block, the third residual block, and the fourth residual block are all 3×3, the convolution step size is 1, the number of input channels is 64, and the number of output channels is 64. The activation function used in the third convolution layer of the first residual block, the second residual block, the third residual block, and the fourth residual block is "ReLU", and the fourth convolution layer does not use an activation function.
[0029] In the step 2, the structures of the first enhanced residual block, the second enhanced residual block and the third enhanced residual block are the same, and they are all composed of a first spatial feature transformation layer, a first spatial angle convolution layer, a second spatial feature transformation layer, a second spatial angle convolution layer and a channel attention layer connected in sequence. The structure of the first spatial feature transformation layer and the second spatial feature transformation layer is the same, and they are all composed of a parallel tenth convolution layer and an eleventh convolution layer. The structure of the first spatial angle convolution layer and the second spatial angle convolution layer is the same, and they are all composed of a twelfth convolution layer and a thirteenth convolution layer connected in sequence. The channel attention layer is composed of a global mean pooling layer, a fourteenth convolution layer and a fifteenth convolution layer connected in sequence;
[0030] The input end of the tenth convolutional layer in the first spatial feature transformation layer in the first enhanced residual block receives F Align,1 All feature maps in the first spatial feature transformation layer in the first enhanced residual block output 64 widths of the tenth convolutional layer. And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the first spatial feature transformation layer in the first enhanced residual block receives F Align,1 For all feature maps in the first spatial feature transformation layer in the first enhanced residual block, the output of the eleventh convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as The input end of the first spatial feature transformation layer in the first enhanced residual block receives F LR All feature maps in F LR All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the first spatial feature transformation layer in the first enhanced residual block. The set of these feature maps is recorded as
[0031] The input end of the twelfth convolutional layer in the first spatial angle convolutional layer in the first enhanced residual block receives For all feature maps in the first enhanced residual block, the output of the twelfth convolutional layer of the first spatial angle convolutional layer outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in the first enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the first spatial angle convolution layer in the first enhanced residual block receives The reorganization operation result of all feature maps in the first enhanced residual block is that the output end of the thirteenth convolution layer of the first spatial angle convolution layer in the first enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the first spatial angle convolution layer in the first enhanced residual block. The set composed of these feature maps is recorded as
[0032] The input end of the tenth convolutional layer in the second spatial feature transformation layer in the first enhanced residual block receives F Align,1 All feature maps in the first enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the second spatial feature transformation layer in the first enhanced residual block outputs 64 frames. And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the second spatial feature transformation layer in the first enhanced residual block receives F Align,1 All feature maps in the first enhanced residual block have 64 outputs at the output end of the eleventh convolutional layer in the second spatial feature transformation layer. And the height is The feature map of the output is recorded as The input end of the second spatial feature transformation layer in the first enhanced residual block receives All feature maps in All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the second spatial feature transformation layer in the first enhanced residual block. The set of these feature maps is recorded as
[0033] The input end of the twelfth convolutional layer in the second spatial angle convolutional layer in the first enhanced residual block receives For all feature maps in the first enhanced residual block, the output of the twelfth convolutional layer of the second spatial angle convolutional layer outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in the first enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the second spatial angle convolution layer in the first enhanced residual block receives The reorganization operation result of all feature maps in the first enhanced residual block is that the output end of the thirteenth convolution layer of the second spatial angle convolution layer in the first enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the second space angle convolution layer in the first enhanced residual block. The set composed of these feature maps is recorded as
[0034] The input of the global mean pooling layer in the channel attention layer in the first enhanced residual block receives All feature maps in the first enhanced residual block have a global mean pooling layer in the channel attention layer, and the output of the global mean pooling layer in the channel attention layer in the first enhanced residual block has a width of 64. And the height is The set of all the output feature maps is recorded as F GAP,1 , F GAP,1 All feature values in each feature map in are the same; the input end of the fourteenth convolutional layer in the channel attention layer in the first enhanced residual block receives F GAP,1 For all feature maps in the first enhanced residual block, the output of the fourteenth convolutional layer in the channel attention layer outputs 4 frames with a width of And the height is The set of all the output feature maps is recorded as F DS,1 ; The input of the fifteenth convolutional layer in the channel attention layer in the first enhanced residual block receives F DS,1 All feature maps in the first enhanced residual block have a width of 64, and the output of the fifteenth convolutional layer in the channel attention layer in the first enhanced residual block has a width of And the height is The set of all the output feature maps is recorded as F US,1 ; F US,1 All feature maps in All feature maps in are multiplied element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the channel attention layer in the first enhanced residual block. The set of these feature maps is recorded as F CA,1 ;
[0035] F CA,1 All feature maps in F LR All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the first enhanced residual block. The set of these feature maps is F En,1 ;
[0036] The input end of the tenth convolutional layer in the first spatial feature transformation layer in the second enhanced residual block receives F Align,2 All feature maps in the second enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the first spatial feature transformation layer in the second enhanced residual block outputs 64 frames. And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the first spatial feature transformation layer in the second enhanced residual block receives F Align,2 All feature maps in the second enhanced residual block output 64 images with a width of And the height is The feature map of the output is recorded as The receiving end of the first spatial feature transformation layer in the second enhanced residual block receives F En,1 All feature maps in F En,1 All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the first spatial feature transformation layer in the second enhanced residual block. The set of these feature maps is recorded as
[0037] The input end of the twelfth convolutional layer in the first spatial angle convolutional layer in the second enhanced residual block receives For all feature maps in the second enhanced residual block, the output of the twelfth convolutional layer of the first spatial angle convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as right All feature maps in the second enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the first spatial angle convolution layer in the second enhanced residual block receives The reorganization operation result of all feature maps in the second enhanced residual block is that the output end of the thirteenth convolution layer of the first spatial angle convolution layer in the second enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the first spatial angle convolution layer in the second enhanced residual block. The set composed of these feature maps is recorded as
[0038] The input end of the tenth convolutional layer in the second spatial feature transformation layer in the second enhanced residual block receives F Align,2 All feature maps in the second spatial feature transformation layer in the second enhanced residual block output 64 widths of the output end of the tenth convolutional layer And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the second spatial feature transformation layer in the second enhanced residual block receives F Align,2 All feature maps in the second spatial feature transformation layer in the second enhanced residual block output 64 widths of the eleventh convolutional layer. And the height is The feature map of the output is recorded as The receiving end receives the second spatial feature transformation layer in the second enhanced residual block All feature maps in All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the second spatial feature transformation layer in the second enhanced residual block. The set of these feature maps is recorded as
[0039] The input end of the twelfth convolution layer in the second spatial angle convolution layer in the second enhanced residual block receives All feature maps in the second enhanced residual block have 64 pixels of width at the output of the twelfth convolution layer of the second spatial angle convolution layer. And the height is The feature map of the output is recorded as right All feature maps in the second enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the second spatial angle convolution layer in the second enhanced residual block receives The reorganization operation result of all feature maps in the second enhanced residual block is that the output end of the thirteenth convolution layer of the second spatial angle convolution layer in the second enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the second spatial angle convolution layer in the second enhanced residual block. The set composed of these feature maps is recorded as
[0040] The input of the global mean pooling layer in the channel attention layer in the second enhanced residual block receives All feature maps in the second enhanced residual block have a global mean pooling layer in the channel attention layer, and the output of the global mean pooling layer in the channel attention layer in the second enhanced residual block has a width of 64. And the height is The set of all the output feature maps is recorded as F GAP,2 , F GAP,2 All feature values in each feature map in are the same; the input end of the fourteenth convolutional layer in the channel attention layer in the second enhanced residual block receives F GAP,2 All feature maps in the second enhanced residual block have 4 outputs at the output of the 14th convolutional layer in the channel attention layer. And the height is The set of all the output feature maps is recorded as F DS,2 ; The input end of the fifteenth convolutional layer in the channel attention layer in the second enhanced residual block receives F DS,2 All feature maps in the second enhanced residual block have a width of 64, and the output of the fifteenth convolutional layer in the channel attention layer in the second enhanced residual block has a width of And the height is The set of all the output feature maps is recorded as F US,2 ; F US,2 All feature maps in All feature maps in are multiplied element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the channel attention layer in the second enhanced residual block. The set of these feature maps is recorded as F CA,2 ;
[0041] F CA,2All feature maps in F En,1 All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the second enhanced residual block. The set of these feature maps is F En,2 ;
[0042] The input end of the tenth convolutional layer in the first spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the first spatial feature transformation layer in the third enhanced residual block outputs 64 And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the first spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have 64 outputs at the output end of the eleventh convolutional layer in the first spatial feature transformation layer. And the height is The feature map of the output is recorded as The receiving end of the first spatial feature transformation layer in the third enhanced residual block receives F En,2 All feature maps in F En,2 All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the first spatial feature transformation layer in the third enhanced residual block. The set of these feature maps is recorded as
[0043] The input end of the twelfth convolutional layer in the first spatial angle convolutional layer in the third enhanced residual block receives For all feature maps in the third enhanced residual block, the output of the twelfth convolutional layer of the first spatial angle convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as right All feature maps in the third enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the first spatial angle convolution layer in the third enhanced residual block receives The reorganization operation result of all feature maps in the third enhanced residual block is that the output end of the thirteenth convolution layer of the first spatial angle convolution layer in the third enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the first space angle convolution layer in the third enhanced residual block. The set composed of these feature maps is recorded as
[0044] The input end of the tenth convolutional layer in the second spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the second spatial feature transformation layer in the third enhanced residual block outputs 64 And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the second spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have 64 outputs with a width of And the height is The feature map of the output is recorded as The receiving end receives the second spatial feature transformation layer in the third enhanced residual block All feature maps in All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the second spatial feature transformation layer in the third enhanced residual block. The set of these feature maps is recorded as
[0045] The input end of the twelfth convolution layer in the second spatial angle convolution layer in the third enhanced residual block receives For all feature maps in the third enhanced residual block, the output of the twelfth convolutional layer of the second spatial angle convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as right All feature maps in the third enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the second spatial angle convolution layer in the third enhanced residual block receives The reorganization operation result of all feature maps in the third enhanced residual block is that the output end of the thirteenth convolution layer of the second spatial angle convolution layer in the third enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the second spatial angle convolution layer in the third enhanced residual block. The set composed of these feature maps is recorded as
[0046] The input of the global mean pooling layer in the channel attention layer in the third enhanced residual block receives All feature maps in the third enhanced residual block have a global mean pooling layer in the channel attention layer, and the output of the global mean pooling layer in the channel attention layer in the third enhanced residual block has a width of 64. And the height is The set of all the output feature maps is recorded as F GAP,3 , F GAP,3 All feature values in each feature map in are the same; the input end of the fourteenth convolutional layer in the channel attention layer in the third enhanced residual block receives F GAP,3 All feature maps in the third enhanced residual block have 4 outputs at the output of the fourteenth convolutional layer in the channel attention layer. And the height is The set of all the output feature maps is recorded as F DS,3 ; The input end of the fifteenth convolutional layer in the channel attention layer in the third enhanced residual block receives F DS,3 All feature maps in the third enhanced residual block have a width of 64, and the output of the fifteenth convolutional layer in the channel attention layer in the third enhanced residual block has a width of And the height is The set of all the output feature maps is recorded as F US,3 ; F US,3 All feature maps in All feature maps in are multiplied element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the channel attention layer in the third enhanced residual block. The set of these feature maps is recorded as F CA,3 ;
[0047] F CA,3 All feature maps in F En,2 All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the third enhanced residual block. The set of these feature maps is F En,3 ;
[0048] In the above, the size of the convolution kernel of the tenth convolution layer and the eleventh convolution layer in the first enhanced residual block, the second enhanced residual block and the third enhanced residual block is 3×3, the convolution step size is 1, the number of input channels is 64, the number of output channels is 64, and no activation function is used. The size of the convolution kernel of the twelfth convolution layer and the thirteenth convolution layer in the first enhanced residual block, the second enhanced residual block and the third enhanced residual block is 3×3, the convolution step size is 1, the number of input channels is 64, the number of output channels is 64, and the activation function used is All are "ReLU". The size of the convolution kernel of the fourteenth convolution layer in the first enhanced residual block, the second enhanced residual block, and the third enhanced residual block is 1×1, the convolution step is 1, the number of input channels is 64, the number of output channels is 4, and the activation function used is "ReLU". The size of the convolution kernel of the fifteenth convolution layer in the first enhanced residual block, the second enhanced residual block, and the third enhanced residual block is 1×1, the convolution step is 1, the number of input channels is 4, the number of output channels is 64, and the activation function used is "Sigmoid".
[0049] Compared with the prior art, the advantages of the present invention are:
[0050] 1) The method of the present invention takes into account that traditional 2D cameras can collect rich spatial information, which can be used as compensation information for reconstructing the spatial resolution of light field images. Therefore, light field images and 2D high-resolution images are used simultaneously. On this basis, an end-to-end convolutional neural network is constructed to make full use of the information of both to reconstruct high spatial resolution light field images, restore detailed texture information, and retain the parallax structure of the reconstruction results.
[0051] 2) In order to establish the connection between the light field image and the 2D high-resolution image, the method of the present invention constructs an aperture-level feature registration module to explore the correlation between the light field image and the 2D high-resolution image in the high-dimensional feature space, and then accurately registers the feature information of the 2D high-resolution image to the light field image; in addition, the method of the present invention uses the constructed light field feature enhancement module to perform multi-level fusion of the high-resolution features obtained by registration with the shallow light field features extracted from the low spatial resolution light field image, so as to effectively generate high spatial resolution light field features, which can then be reconstructed into a high spatial resolution light field image.
[0052] 3) To improve flexibility and practicality, the method of the present invention adopts a pyramid network reconstruction method, which gradually improves the spatial resolution of the light field image and restores texture and details by reconstructing super-resolution results of a specific scale at different pyramid levels, thereby reconstructing multi-scale results (such as 2×, 4× and 8×) in one forward inference; in addition, the method of the present invention adopts a weight sharing strategy at different pyramid levels to effectively reduce the number of parameters of the constructed pyramid network and reduce the training burden. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 It is a flowchart of the overall implementation process of the method of the present invention;
[0054] Figure 2 A schematic diagram of the composition structure of a convolutional neural network, i.e., a spatial super-resolution network, constructed by the method of the present invention;
[0055] Figure 3a A schematic diagram of the composition structure of a light field feature enhancement module in a convolutional neural network, i.e., a spatial super-resolution network, constructed by the method of the present invention;
[0056] Figure 3b A schematic diagram of the composition structure of the first spatial feature transformation layer and the second spatial feature transformation layer in the light field feature enhancement module in the convolutional neural network, i.e., the spatial super-resolution network constructed by the method of the present invention;
[0057] Figure 3c A schematic diagram of the composition structure of the first spatial angle convolution layer and the second spatial angle convolution layer in the light field feature enhancement module in the convolutional neural network constructed by the method of the present invention, i.e., the spatial super-resolution network;
[0058] Figure 3d A schematic diagram of the composition structure of the channel attention layer in the light field feature enhancement module in the convolutional neural network constructed by the method of the present invention, i.e., the spatial super-resolution network;
[0059] Figure 4 A schematic diagram illustrating a pyramid network reconstruction method established by the method of the present invention;
[0060] Figure 5a The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the EPFL light field image database using the bicubic interpolation method. The sub-aperture image under the central coordinate is taken here for display;
[0061] Figure 5b The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the EPFL light field image database using the method of Haris et al. The sub-aperture image under the central coordinate is taken here for display;
[0062] Figure 5c The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the EPFL light field image database using the method of Lai et al. The sub-aperture image under the central coordinate is taken here for display;
[0063] Figure 5dThe reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the EPFL light field image database using the method of Yeung et al. The sub-aperture image under the central coordinate is taken here for display;
[0064] Figure 5e The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the EPFL light field image database using the method of Wang et al. The sub-aperture image under the central coordinate is taken here for display;
[0065] Figure 5f The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the EPFL light field image database using the method of Jin et al. The sub-aperture image under the central coordinate is taken here for display;
[0066] Figure 5g The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the EPFL light field image database using the method of Boominathan et al. The sub-aperture image under the central coordinate is taken here for display;
[0067] Figure 5h The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the EPFL light field image database under test using the method of the present invention. Here, the sub-aperture image under the central coordinate is taken for display;
[0068] Figure 5i The labeled high spatial resolution light field image corresponding to the low spatial resolution light field image in the EPFL light field image database for testing. Here, the sub-aperture image under the central coordinate is taken for display;
[0069] Figure 6a The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the bicubic interpolation method. The sub-aperture image under the central coordinate is taken here for display;
[0070] Figure 6b The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of Haris et al. The sub-aperture image under the central coordinate is taken here for display;
[0071] Figure 6cThe reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of Lai et al. The sub-aperture image under the central coordinate is taken here for display;
[0072] Figure 6d The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of Yeung et al. The sub-aperture image under the central coordinate is taken here for display;
[0073] Figure 6e The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of Wang et al. The sub-aperture image under the central coordinate is taken here for display;
[0074] Figure 6f The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of Jin et al. The sub-aperture image under the central coordinate is taken here for display;
[0075] Figure 6g The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of Boominathan et al. The sub-aperture image under the central coordinate is taken here for display;
[0076] Figure 6h The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of the present invention. Here, the sub-aperture image under the central coordinate is taken for display;
[0077] Figure 6i Labeled high spatial resolution light field images corresponding to the low spatial resolution light field images in the STFLytro light field image database for testing. DETAILED DESCRIPTION
[0078] The present invention is further described in detail below with reference to the accompanying drawings.
[0079] With the development of immersive media and technology, users are increasingly inclined to watch visual content such as images / videos that are interactive and immersive. However, traditional 2D imaging methods can only collect the intensity information of light in the scene, and cannot provide the depth information of the scene. In contrast, 3D imaging technology can obtain more scene information, but the depth information it contains is limited and is generally used for stereoscopic display. Light field imaging, as an emerging imaging technology, can simultaneously collect the intensity and direction information of light in the scene in a single shot, thereby more effectively recording the real world, and is receiving widespread attention. At the same time, some optical instruments and equipment based on light field imaging have been developed to promote the application and development of light field technology. Limited by the size of the imaging sensor, the 4D light field images obtained using a light field camera have the problem of mutual checks and balances between spatial and angular resolution. Simply put, while providing high angular resolution, 4D light field images will inevitably suffer from low spatial resolution, which seriously affects the practical applications of 4D light field images, such as refocusing, depth estimation, etc. To this end, the present invention proposes a light field image spatial super-resolution reconstruction method.
[0080] It uses heterogeneous imaging to obtain a 2D high-resolution image while capturing a light field image, and then uses the captured 2D high-resolution image as supplementary information to help enhance the spatial resolution of the light field image. Specifically, a spatial super-resolution network is constructed, which mainly includes an encoder, an aperture-level feature registration module, a light field feature enhancement module, and a decoder. First, the encoder is used to extract multi-scale features from the upsampled low spatial resolution light field image, the blurred 2D high-resolution image, and the 2D high-resolution image itself. Then, the aperture-level feature registration module is used to learn the 2D high-resolution features and the low-resolution light field features. The correspondence between them is used to register the 2D high-resolution features to each sub-aperture image of the light field image and form the registered high-resolution light field features; then, a light field feature enhancement module is used to enhance the shallow light field features extracted from the input light field image using the registered high-resolution light field features to obtain enhanced high-resolution light field features; finally, a decoder is used to reconstruct the enhanced high-resolution light field features into a high-quality high-spatial resolution light field image; in addition, a pyramid network reconstruction architecture is adopted to reconstruct a high-spatial resolution light field image of a specific upsampling scale at each pyramid level, so that multi-scale reconstruction results can be generated simultaneously.
[0081] The present invention proposes a light field image spatial super-resolution reconstruction method, and its overall implementation flow chart is as follows: Figure 1 As shown, it includes the following steps:
[0082] Step 1: Select Num color three-channel low spatial resolution light field images with a spatial resolution of W×H and an angular resolution of V×U, the corresponding Num color three-channel 2D high-resolution images with a resolution of αW×αH, and the corresponding Num color three-channel reference high spatial resolution light field images with a spatial resolution of αW×αH and an angular resolution of V×U; wherein, Num>1, in this embodiment, Num=200, in this embodiment, W×H is 75×50, V×U is 5×5, α represents the spatial resolution improvement multiple, the value of α is greater than 1, and the value of α is 8 in this embodiment.
[0083] Step 2: Build a convolutional neural network as a spatial super-resolution network: Figure 2 As shown, the spatial super-resolution network includes an encoder for extracting multi-scale features, an aperture-level feature registration module for registering light field features and 2D high-resolution features, a shallow feature extraction layer for extracting shallow features from low spatial resolution light field images, a light field feature enhancement module for fusing light field features and 2D high-resolution features, a spatial attention block for alleviating registration errors in coarse-scale features, and a decoder for reconstructing latent features into a light field image.
[0084] For the encoder, it consists of a first convolutional layer, a second convolutional layer, a first residual block and a second residual block connected in sequence. The input end of the first convolutional layer receives three inputs in parallel, which are a single-channel image L of a low spatial resolution light field image with a spatial resolution of W×H and an angular resolution of V×U. LR The width of the reconstructed image after spatial resolution upsampling is α s W×V and height is α s The H×U sub-aperture image array is denoted as A width of α s W and height is α s The single-channel image of the blurred 2D high-resolution image of H is denoted as and a width of α s W and height is α s A single-channel image of the 2D high-resolution image of H, denoted as I HR , the output of the first convolutional layer is for Output 64 images with a width of α s W×V and height is α s The H×U feature map will be targeted The set of all output feature maps is recorded as The output of the first convolutional layer is Output 64 images with a width of α s W and height is α s The feature map of H will be The set of all output feature maps is recorded as The output of the first convolutional layer is I HR Output 64 images with a width of α s W and height is α s The feature map of H will be HR The set of all output feature maps is recorded as Y HR,0 ; The input end of the second convolutional layer receives three inputs in parallel, namely All feature maps in All feature maps and Y HR,0 All feature maps in the second convolutional layer are Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second convolutional layer is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second convolutional layer is for Y HR,0 Output 64 width And the height is The feature map of Y HR,0 The set of all output feature maps is recorded as Y HR,1 ; The input end of the first residual block receives three inputs in parallel, namely All feature maps in All feature maps and Y HR,1 For all feature maps in , the output of the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the first residual block is for Y HR,1 Output 64 width And the height is The feature map of Y HR,1 The set of all output feature maps is recorded as Y HR,2; The input end of the second residual block receives three inputs in parallel, namely All feature maps in All feature maps and Y HR,2 For all feature maps in , the output of the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second residual block is for Y HR,2 Output 64 width And the height is The feature map of Y HR,2 The set of all output feature maps is recorded as Y HR,3 ;in, is a single-channel image L obtained by transforming a low spatial resolution light field image with a spatial resolution of W×H and an angular resolution of V×U LR The width of the image reconstructed after the existing bicubic interpolation upsampling is α s W×V and height is α s H×U sub-aperture image array, To pass I HR First perform bicubic interpolation downsampling and then perform bicubic interpolation upsampling to obtain, α s represents the spatial resolution sampling factor, in this embodiment α s The value is 2, α s 3 =α, the upsampling factor of bicubic interpolation upsampling and the downsampling factor of bicubic interpolation downsampling are both α s The size of the convolution kernel of the first convolution layer is 3×3, the convolution step is 1, the number of input channels is 1, and the number of output channels is 64. The size of the convolution kernel of the second convolution layer is 3×3, the convolution step is 2, the number of input channels is 64, and the number of output channels is 64. The activation function used in the first and second convolution layers is "ReLU".
[0085] For the aperture-level feature registration module, its input receives three types of feature maps. The first type is All feature maps in the second category are All feature maps in the third category include four inputs, namely Y HR,0 All feature maps in YHR,1 All feature maps in Y HR,2 All feature maps in Y HR,3 All feature maps in the aperture level feature registration module are first All feature maps in Y HR,0 All feature maps in Y HR,1 All feature maps in Y HR,2 All feature maps and Y HR,3 All feature maps in are replicated V×U times, so that All feature maps in Y HR,1 All feature maps in Y HR,2 All feature maps and Y HR,3 The width of all feature maps in becomes And the height becomes Even if the size and and make Y HR,0 The width of all feature maps in becomes α s W×V and the height becomes α s H×U, that is, the size Match the size of the feature map in ; then All feature maps and All feature maps in the existing block matching are performed, and after the block matching is completed, a width of And the height is The coordinate index map of CI ; Then according to P CI , Y HR,1 All feature maps in All feature maps in are spatially aligned to obtain 64 feature maps with a width of And the height is The set of all the obtained registration feature maps is recorded as F Align,1 Similarly, according to P CI , Y HR,2 All feature maps in All feature maps in are spatially aligned to obtain 64 feature maps with a width of And the height is The set of all the obtained registration feature maps is recorded as F Align,2 According to P CI , Y HR,3 All feature maps in All feature maps in are spatially aligned to obtain 64 feature maps with a width of And the height is The set of all the obtained registration feature maps is recorded as FAlign,3 ; Then for P CI Perform bicubic interpolation upsampling to obtain a width of α s W×V and height is α s The coordinate index map of H×U is denoted as Finally, according to Y HR,0 All feature maps in All feature maps in the spatial position are registered to obtain 64 images with a width of α s W×V and height is α s The set of all the obtained registration feature maps is denoted as F Align,0 ; The output of the aperture level feature registration module outputs F Align,0 All feature maps in F Align,1 All feature maps in F Align,2 All feature maps and F Align,3 All feature maps in ; the accuracy measure for block matching is the texture and structure similarity index, the size of the block used for block matching is 3×3, and the upsampling factor of bicubic interpolation upsampling is α s ; Since high-level features describe the similarity of images at the semantic level more compactly and suppress irrelevant textures, this is All feature maps and All feature maps in the block match are performed, and the obtained coordinate index map P CI Reflects The feature maps in The spatial position registration relationship between the feature maps in the convolution operation does not change the spatial position information of the feature map. CI It also reflects The feature maps in The spatial registration relationship between the feature maps in , and The feature maps in The spatial position registration relationship between the feature maps in is obtained after bicubic interpolation upsampling Reflects The feature maps in The spatial position registration relationship between the feature maps in .
[0086] For the shallow feature extraction layer, it consists of a fifth convolutional layer, the input of which receives a single-channel image L of a low spatial resolution light field image with a spatial resolution of W×H and an angular resolution of V×U LRThe reorganized sub-aperture image array is of width W×V and height H×U. The output of the fifth convolutional layer outputs 64 feature maps of width W×V and height H×U. The set of all the output feature maps is denoted as F LR ; Among them, the size of the convolution kernel of the fifth convolution layer is 3×3, the convolution step is 1, the number of input channels is 1, the number of output channels is 64, and the activation function used in the fifth convolution layer is "ReLU".
[0087] For the light field feature enhancement module, such as Figure 3a As shown, it consists of a first enhanced residual block, a second enhanced residual block and a third enhanced residual block connected in sequence, and the input end of the first enhanced residual block receives F Align,1 All feature maps and F LR All feature maps in α s When the value is 2, W×V is equivalent to H×U is equivalent to That is F LR The size of the feature map in F Align,1 The size of the feature map in is the same as that in the first enhanced residual block, and the output end outputs 64 widths. And the height is The set of all the output feature maps is recorded as F En,1 ; The input end of the second enhanced residual block receives F Align,2 All feature maps and F En,1 All feature maps in the second enhanced residual block output 64 images with a width of And the height is The set of all the output feature maps is recorded as F En,2 ; The input end of the third enhanced residual block receives F Align,3 All feature maps and F En,2 All feature maps in the third enhanced residual block output 64 pixels with a width of And the height is The set of all the output feature maps is recorded as F En,3 .
[0088] For the spatial attention block, it consists of the sixth convolutional layer and the seventh convolutional layer connected in sequence. The input end of the sixth convolutional layer receives F Align,0 All feature maps in the , the output of the sixth convolutional layer outputs 64 images with a width of α s W×V and height is α s H×U spatial attention feature map, the set of all output spatial attention feature maps is recorded as F SA1 ; The input of the seventh convolutional layer receives F SA1All spatial attention feature maps in the , the output of the seventh convolutional layer outputs 64 images with a width of α s W×V and height is α s H×U spatial attention feature map, the set of all output spatial attention feature maps is recorded as F SA2 ; F Align,0 All feature maps in F SA2 All spatial attention feature maps in are multiplied element by element, and the set of all feature maps obtained is recorded as F WA,0 ; F WA,0 All feature maps in are used as all feature maps outputted by the output end of the spatial attention block; among them, the convolution kernel sizes of the sixth and seventh convolutional layers are both 3×3, the convolution step sizes are both 1, the number of input channels is both 64, the number of output channels is both 64, the activation function used in the sixth convolutional layer is “ReLU”, and the activation function used in the seventh convolutional layer is “Sigmoid”.
[0089] For the decoder, it consists of the third residual block, the fourth residual block, the sub-pixel convolution layer, the eighth convolution layer and the ninth convolution layer connected in sequence. The input end of the third residual block receives F En,3 All feature maps in the third residual block output 64 images with a width of And the height is The set of all the output feature maps is recorded as F Dec,1 ; The input end of the fourth residual block receives F Dec,1 All feature maps in , the output of the fourth residual block outputs 64 images with a width of And the height is The set of all the output feature maps is recorded as F Dec,2 ; The input of the sub-pixel convolution layer receives F Dec,2 For all feature maps in , the output of the sub-pixel convolutional layer outputs 256 images with a width of And the height is The feature map is 256 images with a width of And the height is The feature map is further converted into 64 images with a width of α s W×V and height is α s The feature map of H×U, the set of all the converted feature maps is recorded as F Dec,3 ; The input of the eighth convolutional layer receives F Dec,3 All feature maps in F WA,0 The result of element-by-element addition of all feature maps in the eighth convolutional layer is 64 images with a width of α. s W×V and height is α sThe feature map of H×U, the set of all the output feature maps is recorded as F Dec,4 ; The input of the ninth convolutional layer receives F Dec,4 All feature maps in the , the output of the ninth convolutional layer outputs a width of α s W×V and height is α s H×U reconstructed single-channel light field image, and the width of the image is α s W×V and height is α s The reconstructed single-channel light field image of H×U is reorganized into a spatial resolution of α s W×α s H and an angular resolution of V × U high spatial resolution single-channel light field image, denoted as L SR ; Among them, the size of the convolution kernel of the sub-pixel convolution layer is 3×3, the convolution step is 1, the number of input channels is 64, and the number of output channels is 256. The size of the convolution kernel of the eighth convolution layer is 3×3, the convolution step is 1, the number of input channels is 64, and the number of output channels is 64. The size of the convolution kernel of the ninth convolution layer is 1×1, the convolution step is 1, the number of input channels is 64, and the number of output channels is 1. The activation functions used in the sub-pixel convolution layer and the eighth convolution layer are both "ReLU", and the ninth convolution layer does not use an activation function.
[0090] Step 3: Perform color space conversion on each low spatial resolution light field image, the corresponding 2D high resolution image, and the corresponding reference high spatial resolution light field image in the training set, that is, convert from RGB color space to YCbCr color space, and extract the Y channel image; then reorganize the Y channel image of each low spatial resolution light field image into a sub-aperture image array with a width of W×V and a height of H×U to represent it; then, the sub-aperture image array reorganized from the Y channel images of all low spatial resolution light field images in the training set, the Y channel image of the corresponding 2D high resolution image, and the Y channel image of the corresponding reference high spatial resolution light field image constitute the training set; then construct a pyramid network and use the training set for training. The specific process is as follows:
[0091] Step 3_1: If Figure 4 As shown in Figure 1, the constructed spatial super-resolution network is replicated three times and cascaded. The weights of each spatial super-resolution network are shared, that is, the parameters are all the same. The overall network composed of three spatial super-resolution networks is defined as a pyramid network. At each pyramid level, the reconstruction scale of the spatial super-resolution network is set to be the same as α s The same value, α s When the value is 2, the spatial resolution of the light field image is increased by 2 times, so the final reconstruction scale can reach 8, that is, α = α s 3 =8.
[0092] Step 3_2: perform two spatial resolution downsampling on the Y channel image of each reference high spatial resolution light field image in the training set, and use the image obtained after downsampling as the label image; perform two spatial resolution downsampling on the Y channel image of each 2D high resolution image in the training set, and use the image obtained after downsampling as the 2D high resolution Y channel image for the first spatial super resolution network in the pyramid network; then, the sub-aperture image array reconstructed from the Y channel images of all low spatial resolution light field images in the training set, the sub-aperture image array reconstructed from the Y channel images of all low spatial resolution light field images in the training set after one spatial resolution upsampling, all the 2D high resolution Y channel images for the first spatial super resolution network in the pyramid network, and all the blurred 2D high resolution Y channel images for the first spatial super resolution network in the pyramid network after one spatial resolution downsampling and one spatial resolution upsampling are input into the first spatial super resolution network in the constructed pyramid network for training, and the α corresponding to the Y channel image of each low spatial resolution light field image in the training set is obtained. s The high spatial resolution Y channel light field image is reconstructed by times; the spatial resolution upsampling and spatial resolution downsampling methods are both bicubic interpolation, and the scales of spatial resolution upsampling and spatial resolution downsampling are the same as α s The values are the same.
[0093] Step 3_3: Perform a single spatial resolution downsampling on the Y channel image of each reference high spatial resolution light field image in the training set, and use the downsampled image as the label image; perform a single spatial resolution downsampling on the Y channel image of each 2D high resolution image in the training set, and use the downsampled image as the 2D high resolution Y channel image for the second spatial super resolution network in the pyramid network; then perform the α corresponding to the Y channel image of all low spatial resolution light field images in the training set on the Y channel image of the training set. s The sub-aperture image array reconstructed from the high spatial resolution Y channel light field image and the α corresponding to the Y channel image of all low spatial resolution light field images in the training set are reconstructed. sThe sub-aperture image array of the reconstructed high spatial resolution Y channel light field image obtained after one spatial resolution upsampling, all 2D high-resolution Y channel images for the second spatial super-resolution network in the pyramid network, and all 2D high-resolution Y channel images for the second spatial super-resolution network in the pyramid network after one spatial resolution downsampling and one spatial resolution upsampling are input into the second spatial super-resolution network in the constructed pyramid network for training, and the α corresponding to the Y channel image of each low spatial resolution light field image in the training set is obtained. s 2 The high spatial resolution Y channel light field image is reconstructed by times; the spatial resolution upsampling and spatial resolution downsampling methods are both bicubic interpolation, and the scales of spatial resolution upsampling and spatial resolution downsampling are the same as α s The values are the same.
[0094] Step 3_4: Take the Y channel image of each reference high spatial resolution light field image in the training set as the label image; take the Y channel image of each 2D high resolution image in the training set as the 2D high resolution Y channel image for the third spatial super resolution network in the pyramid network; then take the α corresponding to the Y channel image of all low spatial resolution light field images in the training set as the label image. s 2 The sub-aperture image array reconstructed from the high spatial resolution Y channel light field image and the α corresponding to the Y channel image of all low spatial resolution light field images in the training set are reconstructed. s 2 The sub-aperture image array of the reconstructed high spatial resolution Y channel light field image obtained after one spatial resolution upsampling, all 2D high-resolution Y channel images for the third spatial super-resolution network in the pyramid network, and all 2D high-resolution Y channel images for the third spatial super-resolution network in the pyramid network after one spatial resolution downsampling and one spatial resolution upsampling are input into the third spatial super-resolution network in the constructed pyramid network for training, and the α corresponding to the Y channel image of each low spatial resolution light field image in the training set is obtained. s 3 The high spatial resolution Y channel light field image is reconstructed by times; the spatial resolution upsampling and spatial resolution downsampling methods are both bicubic interpolation, and the scales of spatial resolution upsampling and spatial resolution downsampling are the same as α s The values are the same.
[0095] After the training, the optimal weight parameters of all convolution kernels in each spatial super-resolution network in the pyramid network are obtained, that is, a well-trained spatial super-resolution network model is obtained; this network model realizes a specific super-resolution reconstruction scale at each pyramid level, so that a multi-scale super-resolution result can be output in one forward inference (i.e., in α s When the value is 2, the scale is 2×, 4× and 8×); in addition, by sharing the weights of the spatial super-resolution network at each pyramid level, the amount of network parameters can be effectively reduced and the training burden can be reduced.
[0096] Step 4: arbitrarily select a color three-channel low spatial resolution light field image and a corresponding color three-channel 2D high resolution image as test images; then convert the color three-channel low spatial resolution light field image and the corresponding color three-channel 2D high resolution image from RGB color space to YCbCr color space, and extract the Y channel image; then reorganize the Y channel image of the low spatial resolution light field image into a sub-aperture image array for representation; then reorganize the sub-aperture image array of the reorganized Y channel image of the low spatial resolution light field image, the sub-aperture image array of the reorganized image after a spatial resolution upsampling of the Y channel image of the low spatial resolution light field image, the Y channel image of the 2D high resolution image, and the Y channel image of the 2D high resolution image after a spatial resolution downsampling and a spatial resolution upsampling to obtain a blurred 2D high resolution Y channel The channel image is input into the spatial super-resolution network model, and the test obtains the reconstructed high spatial resolution Y channel light field image corresponding to the Y channel image of the low spatial resolution light field image; then the Cb channel image and Cr channel image of the low spatial resolution light field image are up-sampled by bicubic interpolation respectively to obtain the reconstructed high spatial resolution Cb channel light field image corresponding to the Cb channel image of the low spatial resolution light field image and the reconstructed high spatial resolution Cr channel light field image corresponding to the Cr channel image of the low spatial resolution light field image; finally, the reconstructed high spatial resolution Y channel light field image, the reconstructed high spatial resolution Cb channel light field image and the reconstructed high spatial resolution Cr channel light field image are cascaded in the color channel dimension, and the cascade result is reconverted to the RGB color space to obtain the reconstructed high spatial resolution light field image of the three color channels corresponding to the low spatial resolution light field image.
[0097] In this embodiment, in step 2, the first residual block, the second residual block, the third residual block and the fourth residual block have the same structure, which are composed of the third convolutional layer and the fourth convolutional layer connected in sequence, and the input end of the third convolutional layer in the first residual block receives three inputs in parallel, which are All feature maps in All feature maps and Y HR,1For all feature maps in , the output of the third convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the first residual block is for Y HR,1 Output 64 width And the height is The feature map of Y HR,1 The set of all output feature maps is recorded as The input end of the fourth convolutional layer in the first residual block receives three inputs in parallel, which are All feature maps in All feature maps and For all feature maps in , the output of the fourth convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the first residual block for All the feature maps outputted, the set of these feature maps is Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the first residual block for All the feature maps outputted, the set of these feature maps is Y HR,1 All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the first residual block for Y HR,1 All the feature maps output, the set of these feature maps is Y HR,2 .
[0098] The input end of the third convolutional layer in the second residual block receives three inputs in parallel, which are All feature maps in All feature maps and Y HR,2 For all feature maps in , the output of the third convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the second residual block is for Y HR,2 Output 64 width And the height is The feature map of Y HR,2 The set of all output feature maps is recorded as The input end of the fourth convolutional layer in the second residual block receives three inputs in parallel, which are All feature maps in All feature maps and For all feature maps in , the output of the fourth convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the second residual block for All the feature maps outputted, the set of these feature maps is Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the second residual block for All the feature maps outputted, the set of these feature maps is Y HR,2 All feature maps in All feature maps in Y are added element by element, and all the obtained feature maps are used as the output of the second residual block for Y HR,2 All the feature maps output, the set of these feature maps is Y HR,3 .
[0099] The input of the third convolutional layer in the third residual block receives F En,3 All feature maps in the third residual block have a width of 64. And the height is The feature map of the output is recorded as The input of the fourth convolutional layer in the third residual block receives All feature maps in the third residual block have a width of 64. And the height is The feature map of the output is recorded as F En,3 All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the third residual block. The set of these feature maps is F Dec,1 .
[0100] The input of the third convolutional layer in the fourth residual block receives F Dec,1 All feature maps in the fourth residual block have a width of 64. And the height is The feature map of the output is recorded as The input end of the fourth convolutional layer in the fourth residual block receives All feature maps in the fourth residual block have a width of 64. And the height is The feature map of the output is recorded as F Dec,1 All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the fourth residual block. The set of these feature maps is F Dec,2 .
[0101] As mentioned above, the size of the convolution kernels of the third convolution layer and the fourth convolution layer in the first residual block, the second residual block, the third residual block, and the fourth residual block are all 3×3, the convolution step size is 1, the number of input channels is 64, and the number of output channels is 64. The activation function used in the third convolution layer of the first residual block, the second residual block, the third residual block, and the fourth residual block is "ReLU", and the fourth convolution layer does not use an activation function.
[0102] In this embodiment, in step 2, Figure 3a , Figure 3b , Figure 3c and Figure 3d As shown, the structures of the first enhanced residual block, the second enhanced residual block and the third enhanced residual block are the same, and they are all composed of a first spatial feature transformation layer, a first spatial angle convolution layer, a second spatial feature transformation layer, a second spatial angle convolution layer and a channel attention layer connected in sequence. The structures of the first spatial feature transformation layer and the second spatial feature transformation layer are the same, and they are both composed of a parallel tenth convolution layer and an eleventh convolution layer. The structures of the first spatial angle convolution layer and the second spatial angle convolution layer are the same, and they are both composed of a twelfth convolution layer and a thirteenth convolution layer connected in sequence. The channel attention layer is composed of a global mean pooling layer, a fourteenth convolution layer and a fifteenth convolution layer connected in sequence.
[0103] The input end of the tenth convolutional layer in the first spatial feature transformation layer in the first enhanced residual block receives F Align,1 All feature maps in the first spatial feature transformation layer in the first enhanced residual block output 64 widths of the tenth convolutional layer. And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the first spatial feature transformation layer in the first enhanced residual block receives F Align,1 For all feature maps in the first spatial feature transformation layer in the first enhanced residual block, the output of the eleventh convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as The input end of the first spatial feature transformation layer in the first enhanced residual block receives F LR All feature maps in F LR All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the first spatial feature transformation layer in the first enhanced residual block. The set of these feature maps is recorded as
[0104] The input end of the twelfth convolutional layer in the first spatial angle convolutional layer in the first enhanced residual block receives For all feature maps in the first enhanced residual block, the output of the twelfth convolutional layer of the first spatial angle convolutional layer outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in the first enhanced residual block are reorganized from the spatial dimension to the angular dimension (the reorganization operation is a conventional processing method for light field images. The reorganization operation only changes the order of each eigenvalue in the feature map, but does not change the size of the eigenvalue). The input end of the thirteenth convolution layer in the first spatial angle convolution layer in the first enhanced residual block receives The reorganization operation result of all feature maps in the first enhanced residual block is that the output end of the thirteenth convolution layer of the first spatial angle convolution layer in the first enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the first spatial angle convolution layer in the first enhanced residual block. The set composed of these feature maps is recorded as
[0105] The input end of the tenth convolutional layer in the second spatial feature transformation layer in the first enhanced residual block receives F Align,1 All feature maps in the first enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the second spatial feature transformation layer in the first enhanced residual block outputs 64 frames. And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the second spatial feature transformation layer in the first enhanced residual block receives F Align,1 All feature maps in the first enhanced residual block have 64 outputs at the output end of the eleventh convolutional layer in the second spatial feature transformation layer. And the height is The feature map of the output is recorded as The input end of the second spatial feature transformation layer in the first enhanced residual block receives All feature maps in All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the second spatial feature transformation layer in the first enhanced residual block. The set of these feature maps is recorded as
[0106] The input end of the twelfth convolutional layer in the second spatial angle convolutional layer in the first enhanced residual block receives For all feature maps in the first enhanced residual block, the output of the twelfth convolutional layer of the second spatial angle convolutional layer outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in the first enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the second spatial angle convolution layer in the first enhanced residual block receives The reorganization operation result of all feature maps in the first enhanced residual block is that the output end of the thirteenth convolution layer of the second spatial angle convolution layer in the first enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the second space angle convolution layer in the first enhanced residual block. The set composed of these feature maps is recorded as
[0107] The input of the global mean pooling layer in the channel attention layer in the first enhanced residual block receives All feature maps in the first enhanced residual block have a global mean pooling layer in the channel attention layer, and the output of the global mean pooling layer in the channel attention layer in the first enhanced residual block has a width of 64. And the height is The set of all the output feature maps is recorded as F GAP,1 , F GAP,1 All the eigenvalues in each feature map in the pool are the same (the global mean pooling layer calculates the global mean for each feature map received at the input independently, thereby converting a feature map into a single eigenvalue, and then copying the obtained eigenvalue to restore the spatial size, that is, copying a single eigenvalue times, the width is And the height is The input of the fourteenth convolutional layer in the channel attention layer in the first enhanced residual block receives F GAP,1 For all feature maps in the first enhanced residual block, the output of the fourteenth convolutional layer in the channel attention layer outputs 4 frames with a width of And the height is The set of all the output feature maps is recorded as F DS,1 ; The input of the fifteenth convolutional layer in the channel attention layer in the first enhanced residual block receives F DS,1 All feature maps in the first enhanced residual block have a width of 64, and the output of the fifteenth convolutional layer in the channel attention layer in the first enhanced residual block has a width of And the height is The set of all the output feature maps is recorded as F US,1 ; F US,1 All feature maps in All feature maps in are multiplied element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the channel attention layer in the first enhanced residual block. The set of these feature maps is recorded as F CA,1 .
[0108] F CA,1 All feature maps in F LR All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the first enhanced residual block. The set of these feature maps is F En,1 .
[0109] The input end of the tenth convolutional layer in the first spatial feature transformation layer in the second enhanced residual block receives F Align,2 All feature maps in the second enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the first spatial feature transformation layer in the second enhanced residual block outputs 64 frames. And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the first spatial feature transformation layer in the second enhanced residual block receives F Align,2All feature maps in the second enhanced residual block output 64 images with a width of And the height is The feature map of the output is recorded as The receiving end of the first spatial feature transformation layer in the second enhanced residual block receives F En,1 All feature maps in F En,1 All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the first spatial feature transformation layer in the second enhanced residual block. The set of these feature maps is recorded as
[0110] The input end of the twelfth convolutional layer in the first spatial angle convolutional layer in the second enhanced residual block receives For all feature maps in the second enhanced residual block, the output of the twelfth convolutional layer of the first spatial angle convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as right All feature maps in the second enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the first spatial angle convolution layer in the second enhanced residual block receives The reorganization operation result of all feature maps in the second enhanced residual block is that the output end of the thirteenth convolution layer of the first spatial angle convolution layer in the second enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the first spatial angle convolution layer in the second enhanced residual block. The set composed of these feature maps is recorded as
[0111] The input end of the tenth convolutional layer in the second spatial feature transformation layer in the second enhanced residual block receives F Align,2 All feature maps in the second spatial feature transformation layer in the second enhanced residual block output 64 widths of the output end of the tenth convolutional layer And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the second spatial feature transformation layer in the second enhanced residual block receives F Align,2 All feature maps in the second spatial feature transformation layer in the second enhanced residual block output 64 widths of the eleventh convolutional layer. And the height is The feature map of the output is recorded as The receiving end receives the second spatial feature transformation layer in the second enhanced residual block All feature maps in All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the second spatial feature transformation layer in the second enhanced residual block. The set of these feature maps is recorded as
[0112] The input end of the twelfth convolution layer in the second spatial angle convolution layer in the second enhanced residual block receives All feature maps in the second enhanced residual block have 64 pixels of width at the output of the twelfth convolution layer of the second spatial angle convolution layer. And the height is The feature map of the output is recorded as right All feature maps in the second enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the second spatial angle convolution layer in the second enhanced residual block receives The reorganization operation result of all feature maps in the second enhanced residual block is that the output end of the thirteenth convolution layer of the second spatial angle convolution layer in the second enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the second spatial angle convolution layer in the second enhanced residual block. The set composed of these feature maps is recorded as
[0113] The input of the global mean pooling layer in the channel attention layer in the second enhanced residual block receives All feature maps in the second enhanced residual block have a global mean pooling layer in the channel attention layer, and the output of the global mean pooling layer in the channel attention layer in the second enhanced residual block has a width of 64. And the height is The set of all the output feature maps is recorded as F GAP,2 , F GAP,2 All feature values in each feature map in are the same; the input end of the fourteenth convolutional layer in the channel attention layer in the second enhanced residual block receives F GAP,2 All feature maps in the second enhanced residual block have 4 outputs at the output of the 14th convolutional layer in the channel attention layer. And the height is The set of all the output feature maps is recorded as F DS,2 ; The input end of the fifteenth convolutional layer in the channel attention layer in the second enhanced residual block receives F DS,2 All feature maps in the second enhanced residual block have a width of 64, and the output of the fifteenth convolutional layer in the channel attention layer in the second enhanced residual block has a width of And the height is The set of all the output feature maps is recorded as F US,2 ; F US,2 All feature maps in All feature maps in are multiplied element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the channel attention layer in the second enhanced residual block. The set of these feature maps is recorded as F CA,2 .
[0114] F CA,2 All feature maps in F En,1 All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the second enhanced residual block. The set of these feature maps is F En,2 .
[0115] The input end of the tenth convolutional layer in the first spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the first spatial feature transformation layer in the third enhanced residual block outputs 64 And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the first spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have 64 outputs at the output end of the eleventh convolutional layer in the first spatial feature transformation layer. And the height is The feature map of the output is recorded as The receiving end of the first spatial feature transformation layer in the third enhanced residual block receives F En,2All feature maps in F En,2 All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the first spatial feature transformation layer in the third enhanced residual block. The set of these feature maps is recorded as
[0116] The input end of the twelfth convolutional layer in the first spatial angle convolutional layer in the third enhanced residual block receives For all feature maps in the third enhanced residual block, the output of the twelfth convolutional layer of the first spatial angle convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as right All feature maps in the third enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the first spatial angle convolution layer in the third enhanced residual block receives The reorganization operation result of all feature maps in the third enhanced residual block is that the output end of the thirteenth convolution layer of the first spatial angle convolution layer in the third enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the first space angle convolution layer in the third enhanced residual block. The set composed of these feature maps is recorded as
[0117] The input end of the tenth convolutional layer in the second spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the second spatial feature transformation layer in the third enhanced residual block outputs 64 And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the second spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have 64 outputs with a width of And the height is The feature map of the output is recorded as The receiving end receives the second spatial feature transformation layer in the third enhanced residual block All feature maps in All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the second spatial feature transformation layer in the third enhanced residual block. The set of these feature maps is recorded as
[0118] The input end of the twelfth convolution layer in the second spatial angle convolution layer in the third enhanced residual block receives For all feature maps in the third enhanced residual block, the output of the twelfth convolutional layer of the second spatial angle convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as right All feature maps in the third enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the second spatial angle convolution layer in the third enhanced residual block receives The reorganization operation result of all feature maps in the third enhanced residual block is that the output end of the thirteenth convolution layer of the second spatial angle convolution layer in the third enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the second spatial angle convolution layer in the third enhanced residual block. The set composed of these feature maps is recorded as
[0119] The input of the global mean pooling layer in the channel attention layer in the third enhanced residual block receives All feature maps in the third enhanced residual block have a global mean pooling layer in the channel attention layer, and the output of the global mean pooling layer in the channel attention layer in the third enhanced residual block has a width of 64. And the height is The set of all the output feature maps is recorded as F GAP,3 , F GAP,3 All feature values in each feature map in are the same; the input end of the fourteenth convolutional layer in the channel attention layer in the third enhanced residual block receives F GAP,3 All feature maps in the third enhanced residual block have 4 outputs at the output of the fourteenth convolutional layer in the channel attention layer. And the height is The set of all the output feature maps is recorded as F DS,3 ; The input end of the fifteenth convolutional layer in the channel attention layer in the third enhanced residual block receives F DS,3 All feature maps in the third enhanced residual block have a width of 64, and the output of the fifteenth convolutional layer in the channel attention layer in the third enhanced residual block has a width of And the height is The set of all the output feature maps is recorded as F US,3 ; F US,3 All feature maps in All feature maps in are multiplied element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the channel attention layer in the third enhanced residual block. The set of these feature maps is recorded as F CA,3 .
[0120] F CA,3 All feature maps in F En,2 All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the third enhanced residual block. The set of these feature maps is F En,3 .
[0121] In the above, the size of the convolution kernel of the tenth convolution layer and the eleventh convolution layer in the first enhanced residual block, the second enhanced residual block and the third enhanced residual block is 3×3, the convolution step size is 1, the number of input channels is 64, the number of output channels is 64, and no activation function is used. The size of the convolution kernel of the twelfth convolution layer and the thirteenth convolution layer in the first enhanced residual block, the second enhanced residual block and the third enhanced residual block is 3×3, the convolution step size is 1, the number of input channels is 64, the number of output channels is 64, and the activation function used is All are "ReLU". The size of the convolution kernel of the fourteenth convolution layer in the first enhanced residual block, the second enhanced residual block, and the third enhanced residual block is 1×1, the convolution step is 1, the number of input channels is 64, the number of output channels is 4, and the activation function used is "ReLU". The size of the convolution kernel of the fifteenth convolution layer in the first enhanced residual block, the second enhanced residual block, and the third enhanced residual block is 1×1, the convolution step is 1, the number of input channels is 4, the number of output channels is 64, and the activation function used is "Sigmoid".
[0122] In order to further illustrate the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method of the present invention.
[0123] The method of the present invention is implemented using the PyTorch deep learning framework. The light field images used for training and testing are all from existing light field image databases, which include real-world scenes and synthetic scenes. These light field image databases can be freely downloaded online. To ensure the reliability and robustness of the test, 200 light field images are randomly selected to form a training image set, and another 70 light field images are selected to form a test image set, wherein the light field images in the training image set and the light field images in the test image set do not overlap with each other. The basic information of the light field image databases used in the training image set and the test image set is shown in Table 1. The four light field image databases, EPFL[1], INRIA[2], STF Lytro[6] and Kalantari et al.[7], are obtained by taking pictures with a Lytro light field camera. Therefore, the light field images obtained are narrow baseline light field data. The STFGantry[5] light field image database is obtained by moving and shooting with a traditional camera fixed on a gantry. Therefore, the light field images obtained have a larger baseline range and belong to wide baseline light field data. The light field images in the HCI new[3] and HCI old[4] light field image databases are artificially synthesized light field images and also belong to wide baseline light field data.
[0124] Table 1 Basic information of the light field image database used in the training image set and the test image set
[0125]
[0126]
[0127] The reference information (or download URL) corresponding to the light field image database used in the training image set and the test image set are as follows:
[0128] [1] Rerabek M, Ebrahimi T. New light field image dataset [C] / / 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX). 2016. (New light field image dataset [C] / / 8th International Conference on Quality of Multimedia Experience, 2016.)
[0129] [2] Pendu ML, Jiang X, Guillemot C. Light Field Inpainting Propagation via Low Rank Matrix Completion[J]. IEEE Transactions on Image Processing, 2018, 27(4): 1981-1993.
[0130] [3] Honauer K, Johannsen O, Kondermann D, et al. A Dataset and Evaluation Methodology for Depth Estimation on 4D Light Fields [C] / / Asian Conference on Computer Vision, 2016. (A Dataset and Evaluation Methodology for Depth Estimation on 4D Light Fields [C] / / Asian Conference on Computer Vision, 2016.)
[0131] [4] Wanner S, Meister S, B Goldluecke. Datasets and Benchmarks for Densely Sampled 4D Light Fields [C] / / International Symposium on Vision Modeling and Visualization, 2013.
[0132] [5] Vaish V, Adams A. The (New) Stanford Light Field Archive, Computer Graphics Laboratory, Stanford University, 2008.
[0133] [6] Raj AS, Lowney M, Shah R, Wetzstein G. Stanford Lytro Light Field Archive, Available: http: / / lightfields.stanford.edu / index.html.
[0134] [7] Kalantari NK, Wang TC, Ramamoorthi R. Learning-Based View Synthesis For Light Field Cameras[J]. ACM Transactions on Graphics, 2016, 35(6): 1-10.
[0135] The light field images in the training image set and the test image set are reorganized into sub-aperture image arrays respectively; considering the vignetting effect of the light field camera (manifested as the low visual quality of the boundary sub-aperture image), the angular resolution of the light field images used for training and testing is cropped to 9×9, that is, only the central high-quality 9×9 view is taken; then the central 5×5 view is taken from the obtained light field image with an angular resolution of 9×9 to form a light field image with an angular resolution of 5×5, and the spatial resolution is downsampled using the bicubic interpolation method with a downsampling scale of 8, that is, the spatial resolution of the light field image is reduced to 1 / 8 of the original light field image, thereby obtaining a low spatial resolution light field image; the original light field image with an angular resolution of 5×5 is used as a reference high spatial resolution light field image (i.e., label image); then a sub-aperture image is selected from the initial 9×9 views (excluding the central 5×5 views) and its resolution is kept unchanged as a 2D high-resolution image. Therefore, the final training set includes a sub-aperture image array reconstructed from the Y channel images of 200 low spatial resolution light field images with an angular resolution of 5×5, the corresponding 200 Y channel images of 2D high resolution images, and the corresponding 200 Y channel images of reference high spatial resolution light field images; the final test set includes a sub-aperture image array reconstructed from the Y channel images of 70 low spatial resolution light field images with an angular resolution of 5×5, the corresponding 70 Y channel images of 2D high resolution images, and the corresponding 70 reference high spatial resolution light field images, among which the 70 reference high spatial resolution light field images do not involve network inference or testing, and are only used for subsequent subjective visual comparison and objective quality evaluation.
[0136] When training the constructed spatial super-resolution network, the parameters of all convolution kernels are initialized using the MSRA initializer; the loss function uses a combination of pixel-domain L1 norm loss and gradient loss; the network is trained using the ADAM optimizer; first, 10 -4 The learning rate is used to train the encoder and decoder parts of the spatial super-resolution network until they converge to a certain extent, and then the learning rate is set to 10 -4 , to train the entire spatial super-resolution network, and the learning rate is decayed by a scaling factor of 0.5 after 25 epochs of training.
[0137] In order to illustrate the performance of the method of the present invention, the method of the present invention is compared with the existing bicubic interpolation method and six existing image super-resolution reconstruction methods, namely, the method based on deep back projection network proposed by Haris et al., the method based on deep Laplacian pyramid network proposed by Lai et al., the method based on space-angle separable convolution proposed by Yeung et al., the method based on space-angle interaction network proposed by Wang et al., the method based on two-stage network proposed by Jin et al. and the method based on mixed input proposed by Boominathan et al., among which, the method of Haris et al. and the method of Lai et al. belong to 2D image super-resolution reconstruction methods (which are independently applied to each sub-aperture image of the light field image), the method of Yeung et al., the method of Wang et al. and the method of Jin et al. belong to ordinary light field image spatial super-resolution reconstruction methods, and the method of Boominathan et al. belongs to a light field image spatial super-resolution reconstruction method using mixed input.
[0138] The objective quality evaluation indicators used here include PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index), and an advanced light field image objective quality evaluation indicator (see Min X, Zhou J, Zhai G, et al. A Metric for Light Field Reconstruction, Compression, and Display Quality Evaluation [J]. IEEE Transactions on Image Processing, 2020, 29: 3790-3804. (A metric for light field reconstruction, compression, and display quality assessment, IEEE Transactions on Image Processing, 2020, 29: 3790-3804.)), where PSNR evaluates the objective quality of super-resolution reconstructed images from the perspective of pixel reconstruction error. The higher the value, the better the image quality; SSIM evaluates the objective quality of super-resolution reconstructed images from the perspective of visual perception. Its value is between 0 and 1. The higher the value, the better the image quality; the light field image objective quality evaluation index effectively evaluates the objective quality of super-resolution reconstructed images by jointly measuring the spatial quality (texture and details) and angular quality (parallax structure) of the light field image. The higher the value, the better the image quality.
[0139] Table 2 shows the comparison of the PSNR (dB) index between the method of the present invention and the existing bicubic interpolation method and the existing light field image spatial super-resolution reconstruction method, Table 3 shows the comparison of the SSIM index between the method of the present invention and the existing bicubic interpolation method and the existing light field image spatial super-resolution reconstruction method, and Table 4 shows the comparison of the objective quality evaluation index of the light field image between the method of the present invention and the existing bicubic interpolation method and the existing light field image spatial super-resolution reconstruction method. It can be seen from the objective data listed in Tables 2, 3 and 4 that, compared with the existing light field image spatial super-resolution reconstruction method (including 2D image super-resolution reconstruction method), the method of the present invention has achieved higher quality scores in the three objective quality evaluation indicators used, and is significantly higher than all the comparison methods, which indicates that the method of the present invention can effectively reconstruct the texture and detail information of the light field image, and restore a good parallax structure; in particular, for light field image databases with different baseline ranges and scene contents, the method of the present invention has achieved the best super-resolution reconstruction effect, which indicates that the method of the present invention can well process narrow baseline and wide baseline light field data, and has good robustness to scene content.
[0140] Table 2 Comparison of the PSNR (dB) index between the method of the present invention and the existing bicubic interpolation method and the existing light field image spatial super-resolution reconstruction method
[0141]
[0142]
[0143] Table 3 Comparison of the SSIM index between the method of the present invention and the existing bicubic interpolation method and the existing light field image spatial super-resolution reconstruction method
[0144]
[0145]
[0146] Table 4 Comparison of objective quality evaluation indicators of light field images using the method of the present invention, the existing bicubic interpolation method, and the existing light field image spatial super-resolution reconstruction method
[0147]
[0148]
[0149] Figure 5a The reconstructed high spatial resolution light field image is given by processing the low spatial resolution light field image in the EPFL light field image database using the bicubic interpolation method. Here, the sub-aperture image under the central coordinate is taken for display; Figure 5b The reconstructed high spatial resolution light field image obtained by processing the low spatial resolution light field image in the EPFL light field image database using the method of Haris et al. is presented. Here, the sub-aperture image under the central coordinate is taken for display; Figure 5c The reconstructed high spatial resolution light field image obtained by processing the low spatial resolution light field image in the EPFL light field image database using the method of Lai et al. is presented. Here, the sub-aperture image under the central coordinate is taken for display; Figure 5d The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the EPFL light field image database using the method of Yeung et al. Here, the sub-aperture image under the central coordinate is taken for display; Figure 5e The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the EPFL light field image database using the method of Wang et al. Here, the sub-aperture image under the central coordinate is taken for display; Figure 5fThe reconstructed high spatial resolution light field image is given by processing the low spatial resolution light field image in the EPFL light field image database using the method of Jin et al. Here, the sub-aperture image under the central coordinate is taken for display; Figure 5g The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the EPFL light field image database using the method of Boominathan et al. Here, the sub-aperture image under the central coordinate is taken for display; Figure 5h The reconstructed high spatial resolution light field image obtained by processing the low spatial resolution light field image in the EPFL light field image database tested by the method of the present invention is given, and the sub-aperture image under the central coordinate is taken here for display; Figure 5i The labeled high spatial resolution light field image corresponding to the low spatial resolution light field image in the tested EPFL light field image database is given. The sub-aperture image under the central coordinate is taken here for display.
[0150] Figure 6a The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the bicubic interpolation method. Here, the sub-aperture image under the central coordinate is taken for display; Figure 6b The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of Haris et al. Here, the sub-aperture image under the central coordinate is taken for display; Figure 6c The reconstructed high spatial resolution light field image obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of Lai et al. is presented. Here, the sub-aperture image under the central coordinate is taken for display; Figure 6d The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the STFLytro light field image database using the method of Yeung et al. Here, the sub-aperture image under the central coordinate is taken for display; Figure 6e The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of Wang et al. Here, the sub-aperture image under the central coordinate is taken for display; Figure 6f The reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the STFLytro light field image database using the method of Jin et al. Here, the sub-aperture image under the central coordinate is taken for display; Figure 6gThe reconstructed high spatial resolution light field image is obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of Boominathan et al. Here, the sub-aperture image under the central coordinate is taken for display; Figure 6h The reconstructed high spatial resolution light field image obtained by processing the low spatial resolution light field image in the tested STFLytro light field image database using the method of the present invention is given, and the sub-aperture image under the central coordinate is taken here for display; Figure 6i The labeled high spatial resolution light field image corresponding to the low spatial resolution light field image in the tested STFLytro light field image database is given. The sub-aperture image under the central coordinate is taken here for display.
[0151] Respectively Figure 5a to Figure 5h and Figure 5i to compare and Figure 6a to Figure 6h and Figure 6i By comparison, it can be clearly seen that the high spatial resolution light field images reconstructed using the existing light field image spatial super-resolution reconstruction methods, including 2D image super-resolution reconstruction methods, cannot restore the texture and detail information of the image. Figure 5a to Figure 5f The lower left rectangular box in the magnified area, and Figures 6a to 6f As shown in the enlarged area of the lower right rectangular box in ; the light field image spatial super-resolution reconstruction method using mixed input has achieved relatively better results, but still contains some blurring artifacts, such as Figure 5g The lower left rectangular box enlarges the area and Figure 6g In contrast, the high spatial resolution light field image reconstructed by the method of the present invention has clear texture and rich details, and is close to the labeled high spatial resolution light field image (i.e. Figure 5i and Figure 6i ), which shows that the method of the present invention can effectively restore the texture information of the light field image. In addition, by reconstructing each sub-aperture image with high quality, the method of the present invention can well ensure the parallax structure of the high spatial resolution light field image finally reconstructed.
[0152] The innovation of the method of the present invention is mainly reflected as follows: first, heterogeneous imaging is used to obtain rich 2D spatial information while capturing high-dimensional light field data, that is, a light field image and a 2D high-resolution image are captured at the same time, and then the spatial resolution of the light field image is effectively improved by using the information of the 2D high-resolution image, and the corresponding texture and details are restored; second, in order to establish and explore the connection between the light field image and the 2D high-resolution image, the method of the present invention constructs an aperture-level feature registration module and a light field feature enhancement module respectively. The former can accurately register the 2D high-resolution information with the 4D light field image information, and the latter uses the high-resolution feature information obtained by the registration to consistently enhance the visual information in the light field feature on this basis, so as to obtain the enhanced high-resolution light field feature; third, a flexible pyramid reconstruction method is adopted, that is, a reconstruction strategy from coarse to fine is used to gradually improve the spatial resolution of the light field image and restore the accurate parallax structure, so that the multi-scale super-resolution result can be reconstructed in one forward inference. In addition, in order to reduce the parameter amount and training burden of the pyramid network, weight sharing is performed at each pyramid level.
Claims
1. A light field image spatial super-resolution reconstruction method, characterized in that The following steps are involved: Step 1: select Num color three-channel low spatial resolution light field images with a spatial resolution of W×H and an angular resolution of V×U, the corresponding Num color three-channel 2D high-resolution images with a resolution of αW×αH, and the corresponding Num color three-channel reference high spatial resolution light field images with a spatial resolution of αW×αH and an angular resolution of V×U; wherein Num>1, α represents the spatial resolution improvement multiple, and the value of α is greater than 1; Step 2: Build a convolutional neural network as a spatial super-resolution network: The spatial super-resolution network includes an encoder for extracting multi-scale features, an aperture-level feature registration module for registering light field features and 2D high-resolution features, a shallow feature extraction layer for extracting shallow features from low spatial resolution light field images, a light field feature enhancement module for fusing light field features and 2D high-resolution features, a spatial attention block for alleviating registration errors in coarse-scale features, and a decoder for reconstructing latent features into a light field image; For the encoder, it consists of a first convolutional layer, a second convolutional layer, a first residual block and a second residual block connected in sequence. The input end of the first convolutional layer receives three inputs in parallel, which are a single-channel image L of a low spatial resolution light field image with a spatial resolution of W×H and an angular resolution of V×U. LR The width of the reconstructed image after spatial resolution upsampling is α s W×V and height is α s The H×U sub-aperture image array is denoted as A width of α s W and height is α s The single-channel image of the blurred 2D high-resolution image of H is denoted as and a width of α s W and height is α s A single-channel image of the 2D high-resolution image of H, denoted as I HR , the output of the first convolutional layer is for Output 64 images with a width of α s W×V and height is α s The H×U feature map will be targeted The set of all output feature maps is recorded as The output of the first convolutional layer is Output 64 images with a width of α s W and height is α s The feature map of H will be The set of all output feature maps is recorded as The output of the first convolutional layer is I HR Output 64 images with a width of α s W and height is α s The feature map of H will be HR The set of all output feature maps is recorded as Y HR,0 ; The input end of the second convolutional layer receives three inputs in parallel, namely All feature maps in All feature maps and Y HR,0 All feature maps in the second convolutional layer are Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second convolutional layer is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second convolutional layer is for Y HR,0 Output 64 width And the height is The feature map of Y HR ,0 The set of all output feature maps is recorded as Y HR,1 ; The input end of the first residual block receives three inputs in parallel, namely All feature maps in All feature maps and Y HR,1 For all feature maps in , the output of the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the first residual block is for Y HR,1 Output 64 width And the height is The feature map of Y HR,1 The set of all output feature maps is recorded as Y HR,2 ; The input end of the second residual block receives three inputs in parallel, namely All feature maps in All feature maps and Y HR,2 For all feature maps in , the output of the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the second residual block is for Y HR,2 Output 64 width And the height is The feature map of Y HR,2 The set of all output feature maps is recorded as Y HR,3 ;in, is a single-channel image L obtained by transforming a low spatial resolution light field image with a spatial resolution of W×H and an angular resolution of V×U LR The width of the reconstructed image after bicubic interpolation upsampling is α s W×V and height is α s H×U sub-aperture image array, To pass I HR First perform bicubic interpolation downsampling and then perform bicubic interpolation upsampling to obtain, α s represents the spatial resolution sampling factor, α s 3 =α, the upsampling factor of bicubic interpolation upsampling and the downsampling factor of bicubic interpolation downsampling are both α s The size of the convolution kernel of the first convolution layer is 3×3, the convolution step is 1, the number of input channels is 1, and the number of output channels is 64. The size of the convolution kernel of the second convolution layer is 3×3, the convolution step is 2, the number of input channels is 64, and the number of output channels is 64. The activation function used in the first and second convolution layers is "ReLU"; For the aperture-level feature registration module, its input receives three types of feature maps. The first type is All feature maps in the second category are All feature maps in the third category include four inputs, namely Y HR,0 All feature maps in Y HR,1 All feature maps in Y HR,2 All feature maps in Y HR,3 All feature maps in the aperture level feature registration module are first All feature maps in Y HR,0 All feature maps in Y HR,1 All feature maps in Y HR,2 All feature maps and Y HR,3 All feature maps in are replicated V×U times, so that All feature maps in Y HR,1 All feature maps in Y HR,2 All feature maps and Y HR,3 The width of all feature maps in becomes And the height becomes Even if the size and and make Y HR,0 The width of all feature maps in becomes α s W×V and the height becomes α s H×U, that is, the size Match the size of the feature map in ; then All feature maps and All feature maps in are block matched, and after the block matching is completed, a width of And the height is The coordinate index map of CI ; Then according to P CI , Y HR,1 All feature maps in All feature maps in are spatially aligned to obtain 64 feature maps with a width of And the height is The set of all the obtained registration feature maps is recorded as F Align,1 Similarly, according to P CI , Y HR,2 All feature maps in All feature maps in are spatially aligned to obtain 64 feature maps with a width of And the height is The set of all the obtained registration feature maps is recorded as F Align,2 According to P CI , Y HR,3 All feature maps in All feature maps in are spatially aligned to obtain 64 feature maps with a width of And the height is The set of all the obtained registration feature maps is recorded as F Align,3 ; Then for P CI Perform bicubic interpolation upsampling to obtain a width of α s W×V and height is α s The coordinate index map of H×U is denoted as Finally, according to Y HR,0 All feature maps in All feature maps in the spatial position are registered to obtain 64 images with a width of α s W×V and height is α s The set of all the obtained registration feature maps is denoted as F Align,0 ; The output of the aperture level feature registration module outputs F Align,0 All feature maps in F Align,1 All feature maps in F Align,2 All feature maps and F Align,3 All feature maps in ; the accuracy measure for block matching is the texture and structure similarity index, the size of the block used for block matching is 3×3, and the upsampling factor of bicubic interpolation upsampling is α s ; For the shallow feature extraction layer, it consists of a fifth convolutional layer, the input of which receives a single-channel image L of a low spatial resolution light field image with a spatial resolution of W×H and an angular resolution of V×U LR The reorganized sub-aperture image array is of width W×V and height H×U. The output of the fifth convolutional layer outputs 64 feature maps of width W×V and height H×U. The set of all the output feature maps is denoted as F LR ; Among them, the size of the convolution kernel of the fifth convolution layer is 3×3, the convolution step is 1, the number of input channels is 1, the number of output channels is 64, and the activation function used in the fifth convolution layer is "ReLU"; For the light field feature enhancement module, it is composed of a first enhanced residual block, a second enhanced residual block and a third enhanced residual block connected in sequence, and the input end of the first enhanced residual block receives F Align,1 All feature maps and F LR All feature maps in the first enhanced residual block have a width of 64. And the height is The set of all the output feature maps is recorded as F En,1 ; The input end of the second enhanced residual block receives F Align,2 All feature maps and F En,1 All feature maps in the second enhanced residual block output 64 images with a width of And the height is The set of all the output feature maps is recorded as F En,2 ; The input end of the third enhanced residual block receives F Align,3 All feature maps and F En,2 All feature maps in the third enhanced residual block output 64 pixels with a width of And the height is The set of all the output feature maps is recorded as F En,3 ; For the spatial attention block, it consists of the sixth convolutional layer and the seventh convolutional layer connected in sequence. The input end of the sixth convolutional layer receives F Align,0 All feature maps in the , the output of the sixth convolutional layer outputs 64 images with a width of α s W×V and height is α s H×U spatial attention feature map, the set of all output spatial attention feature maps is recorded as F SA1 ; The input of the seventh convolutional layer receives F SA1 All spatial attention feature maps in the , the output of the seventh convolutional layer outputs 64 images with a width of α s W×V and height is α s H×U spatial attention feature map, the set of all output spatial attention feature maps is recorded as F SA2 ; F Align,0 All feature maps in F SA2 All spatial attention feature maps in are multiplied element by element, and the set of all feature maps obtained is recorded as F WA,0 ; F WA,0 All feature maps in are used as all feature maps outputted by the output of the spatial attention block; the convolution kernel size of the sixth and seventh convolution layers is 3×3, the convolution step size is 1, the number of input channels is 64, the number of output channels is 64, the activation function used in the sixth convolution layer is "ReLU", and the activation function used in the seventh convolution layer is "Sigmoid"; For the decoder, it consists of the third residual block, the fourth residual block, the sub-pixel convolution layer, the eighth convolution layer and the ninth convolution layer connected in sequence. The input end of the third residual block receives F En,3 All feature maps in the third residual block output 64 images with a width of And the height is The set of all the output feature maps is recorded as F Dec,1 ; The input end of the fourth residual block receives F Dec,1 All feature maps in , the output of the fourth residual block outputs 64 images with a width of And the height is The set of all the output feature maps is recorded as F Dec,2 ; The input of the sub-pixel convolution layer receives F Dec,2 For all feature maps in , the output of the sub-pixel convolutional layer outputs 256 images with a width of And the height is The feature map is 256 images with a width of And the height is The feature map is further converted into 64 images with a width of α s W×V and height is α s The feature map of H×U, the set of all the converted feature maps is recorded as F Dec,3 ; The input of the eighth convolutional layer receives F Dec,3 All feature maps in F WA,0 The result of element-by-element addition of all feature maps in the eighth convolutional layer is 64 images with a width of α. s W×V and height is α s The feature map of H×U, the set of all the output feature maps is recorded as F Dec,4 ; The input of the ninth convolutional layer receives F Dec,4 All feature maps in the , the output of the ninth convolutional layer outputs a width of α s W×V and height is α s H×U reconstructed single-channel light field image, and the width of the image is α s W×V and height is α s The reconstructed single-channel light field image of H×U is reorganized into a spatial resolution of α s W×α s H and an angular resolution of V × U high spatial resolution single-channel light field image, denoted as L SR ; Among them, the size of the convolution kernel of the sub-pixel convolution layer is 3×3, the convolution step is 1, the number of input channels is 64, and the number of output channels is 256. The size of the convolution kernel of the eighth convolution layer is 3×3, the convolution step is 1, the number of input channels is 64, and the number of output channels is 64. The size of the convolution kernel of the ninth convolution layer is 1×1, the convolution step is 1, the number of input channels is 64, and the number of output channels is 1. The activation function used by the sub-pixel convolution layer and the eighth convolution layer is "ReLU", and the ninth convolution layer does not use an activation function; Step 3: Perform color space conversion on each low spatial resolution light field image, the corresponding 2D high resolution image, and the corresponding reference high spatial resolution light field image in the training set, that is, convert from RGB color space to YCbCr color space, and extract the Y channel image; then reorganize the Y channel image of each low spatial resolution light field image into a sub-aperture image array with a width of W×V and a height of H×U to represent it; then, the sub-aperture image array reorganized from the Y channel images of all low spatial resolution light field images in the training set, the Y channel image of the corresponding 2D high resolution image, and the Y channel image of the corresponding reference high spatial resolution light field image constitute the training set; then construct a pyramid network and use the training set for training. The specific process is as follows: Step 3_1: The constructed spatial super-resolution network is replicated three times and cascaded. The weights of each spatial super-resolution network are shared, that is, the parameters are all the same. The overall network composed of three spatial super-resolution networks is defined as a pyramid network; at each pyramid level, the reconstruction scale of the spatial super-resolution network is set to be the same as α s The values are the same; Step 3_2: perform two spatial resolution downsampling on the Y channel image of each reference high spatial resolution light field image in the training set, and use the image obtained after downsampling as the label image; perform two spatial resolution downsampling on the Y channel image of each 2D high resolution image in the training set, and use the image obtained after downsampling as the 2D high resolution Y channel image for the first spatial super resolution network in the pyramid network; then, the sub-aperture image array reconstructed from the Y channel images of all low spatial resolution light field images in the training set, the sub-aperture image array reconstructed from the Y channel images of all low spatial resolution light field images in the training set after one spatial resolution upsampling, all the 2D high resolution Y channel images for the first spatial super resolution network in the pyramid network, and all the blurred 2D high resolution Y channel images for the first spatial super resolution network in the pyramid network after one spatial resolution downsampling and one spatial resolution upsampling are input into the first spatial super resolution network in the constructed pyramid network for training, and the α corresponding to the Y channel image of each low spatial resolution light field image in the training set is obtained. s The high spatial resolution Y channel light field image is reconstructed by times; the spatial resolution upsampling and spatial resolution downsampling methods are both bicubic interpolation, and the scales of spatial resolution upsampling and spatial resolution downsampling are the same as α s The values are the same; Step 3_3: Perform a single spatial resolution downsampling on the Y channel image of each reference high spatial resolution light field image in the training set, and use the downsampled image as the label image; perform a single spatial resolution downsampling on the Y channel image of each 2D high resolution image in the training set, and use the downsampled image as the 2D high resolution Y channel image for the second spatial super resolution network in the pyramid network; then perform the α corresponding to the Y channel image of all low spatial resolution light field images in the training set on the Y channel image of the training set. s The sub-aperture image array reconstructed from the high spatial resolution Y channel light field image and the α corresponding to the Y channel image of all low spatial resolution light field images in the training set are reconstructed. s The sub-aperture image array of the reconstructed high spatial resolution Y channel light field image obtained after one spatial resolution upsampling, all 2D high-resolution Y channel images for the second spatial super-resolution network in the pyramid network, and all 2D high-resolution Y channel images for the second spatial super-resolution network in the pyramid network after one spatial resolution downsampling and one spatial resolution upsampling are input into the second spatial super-resolution network in the constructed pyramid network for training, and the α corresponding to the Y channel image of each low spatial resolution light field image in the training set is obtained. s 2 The high spatial resolution Y channel light field image is reconstructed by times; the spatial resolution upsampling and spatial resolution downsampling methods are both bicubic interpolation, and the scales of spatial resolution upsampling and spatial resolution downsampling are the same as α s The values are the same; Step 3_4: Take the Y channel image of each reference high spatial resolution light field image in the training set as the label image; take the Y channel image of each 2D high resolution image in the training set as the 2D high resolution Y channel image for the third spatial super resolution network in the pyramid network; then take the α corresponding to the Y channel image of all low spatial resolution light field images in the training set as the label image. s 2 The sub-aperture image array reconstructed from the high spatial resolution Y channel light field image and the α corresponding to the Y channel image of all low spatial resolution light field images in the training set are reconstructed. s 2 The sub-aperture image array of the reconstructed high spatial resolution Y channel light field image obtained after one spatial resolution upsampling, all 2D high-resolution Y channel images for the third spatial super-resolution network in the pyramid network, and all 2D high-resolution Y channel images for the third spatial super-resolution network in the pyramid network after one spatial resolution downsampling and one spatial resolution upsampling are input into the third spatial super-resolution network in the constructed pyramid network for training, and the α corresponding to the Y channel image of each low spatial resolution light field image in the training set is obtained. s 3 The high spatial resolution Y channel light field image is reconstructed by times; the spatial resolution upsampling and spatial resolution downsampling methods are both bicubic interpolation, and the scales of spatial resolution upsampling and spatial resolution downsampling are the same as α s The values are the same; After the training is completed, the optimal weight parameters of all convolution kernels in each spatial super-resolution network in the pyramid network are obtained, that is, a well-trained spatial super-resolution network model is obtained; Step 4: arbitrarily select a color three-channel low spatial resolution light field image and a corresponding color three-channel 2D high resolution image as test images; then convert the color three-channel low spatial resolution light field image and the corresponding color three-channel 2D high resolution image from RGB color space to YCbCr color space, and extract the Y channel image; then reorganize the Y channel image of the low spatial resolution light field image into a sub-aperture image array for representation; then reorganize the sub-aperture image array of the reorganized Y channel image of the low spatial resolution light field image, the sub-aperture image array of the reorganized image after a spatial resolution upsampling of the Y channel image of the low spatial resolution light field image, the Y channel image of the 2D high resolution image, and the Y channel image of the 2D high resolution image after a spatial resolution downsampling and a spatial resolution upsampling to obtain a blurred 2D high resolution Y channel The channel image is input into the spatial super-resolution network model, and the test obtains the reconstructed high spatial resolution Y channel light field image corresponding to the Y channel image of the low spatial resolution light field image; then the Cb channel image and Cr channel image of the low spatial resolution light field image are up-sampled by bicubic interpolation respectively to obtain the reconstructed high spatial resolution Cb channel light field image corresponding to the Cb channel image of the low spatial resolution light field image and the reconstructed high spatial resolution Cr channel light field image corresponding to the Cr channel image of the low spatial resolution light field image; finally, the reconstructed high spatial resolution Y channel light field image, the reconstructed high spatial resolution Cb channel light field image and the reconstructed high spatial resolution Cr channel light field image are cascaded in the color channel dimension, and the cascade result is reconverted to the RGB color space to obtain the reconstructed high spatial resolution light field image of the three color channels corresponding to the low spatial resolution light field image.
2. The method for spatial super-resolution reconstruction of a light field image according to claim 1, characterized in that In the step 2, the first residual block, the second residual block, the third residual block and the fourth residual block have the same structure, which are composed of a third convolutional layer and a fourth convolutional layer connected in sequence, and the input end of the third convolutional layer in the first residual block receives three inputs in parallel, which are All feature maps in All feature maps and Y HR,1 For all feature maps in , the output of the third convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the first residual block is for Y HR,1 Output 64 width And the height is The feature map of Y HR,1 The set of all output feature maps is recorded as The input end of the fourth convolutional layer in the first residual block receives three inputs in parallel, which are All feature maps in All feature maps and For all feature maps in , the output of the fourth convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the first residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the first residual block for All the feature maps outputted, the set of these feature maps is Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the first residual block for All the feature maps outputted, the set of these feature maps is Y HR,1 All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the first residual block for Y HR,1 All the feature maps output, the set of these feature maps is Y HR,2 ; The input end of the third convolutional layer in the second residual block receives three inputs in parallel, which are All feature maps in All feature maps and Y HR,2 For all feature maps in , the output of the third convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the third convolutional layer in the second residual block is for Y HR,2 Output 64 width And the height is The feature map of Y HR,2 The set of all output feature maps is recorded as The input end of the fourth convolutional layer in the second residual block receives three inputs in parallel, which are All feature maps in All feature maps and For all feature maps in , the output of the fourth convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as The output of the fourth convolutional layer in the second residual block is Output 64 width And the height is The feature map will be The set of all output feature maps is recorded as Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the second residual block for All the feature maps outputted, the set of these feature maps is Will All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as the output of the second residual block for All the feature maps outputted, the set of these feature maps is Y HR,2 All feature maps in All feature maps in Y are added element by element, and all the obtained feature maps are used as the output of the second residual block for Y HR,2 All the feature maps output, the set of these feature maps is Y HR,3 ; The input of the third convolutional layer in the third residual block receives F En,3 All feature maps in the third residual block have a width of 64. And the height is The feature map of the output is recorded as The input of the fourth convolutional layer in the third residual block receives All feature maps in the third residual block have a width of 64. And the height is The feature map of the output is recorded as F En,3 All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the third residual block. The set of these feature maps is F Dec,1 ; The input of the third convolutional layer in the fourth residual block receives F Dec,1 All feature maps in the fourth residual block have a width of 64. And the height is The feature map of the output is recorded as The input end of the fourth convolutional layer in the fourth residual block receives All feature maps in the fourth residual block have a width of 64. And the height is The feature map of the output is recorded as F Dec,1 All feature maps in All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the fourth residual block. The set of these feature maps is F Dec,2 ; Among them, the size of the convolution kernel of the third convolution layer and the fourth convolution layer in the first residual block, the second residual block, the third residual block and the fourth residual block are all 3×3, the convolution step size is 1, the number of input channels is 64, and the number of output channels is 64. The activation function used in the third convolution layer of the first residual block, the second residual block, the third residual block and the fourth residual block is "ReLU", and the fourth convolution layer does not use an activation function.
3. A light field image spatial super-resolution reconstruction method according to claim 1 or 2, characterized in that In the step 2, the structures of the first enhanced residual block, the second enhanced residual block and the third enhanced residual block are the same, and they are all composed of a first spatial feature transformation layer, a first spatial angle convolution layer, a second spatial feature transformation layer, a second spatial angle convolution layer and a channel attention layer connected in sequence. The structure of the first spatial feature transformation layer and the second spatial feature transformation layer is the same, and they are all composed of a parallel tenth convolution layer and an eleventh convolution layer. The structure of the first spatial angle convolution layer and the second spatial angle convolution layer is the same, and they are all composed of a twelfth convolution layer and a thirteenth convolution layer connected in sequence. The channel attention layer is composed of a global mean pooling layer, a fourteenth convolution layer and a fifteenth convolution layer connected in sequence; The input end of the tenth convolutional layer in the first spatial feature transformation layer in the first enhanced residual block receives F Align,1 All feature maps in the first spatial feature transformation layer in the first enhanced residual block output 64 widths of the tenth convolutional layer. And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the first spatial feature transformation layer in the first enhanced residual block receives F Align,1 For all feature maps in the first spatial feature transformation layer in the first enhanced residual block, the output of the eleventh convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as The input end of the first spatial feature transformation layer in the first enhanced residual block receives F LR All feature maps in F LR All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the first spatial feature transformation layer in the first enhanced residual block. The set of these feature maps is recorded as The input end of the twelfth convolutional layer in the first spatial angle convolutional layer in the first enhanced residual block receives For all feature maps in the first enhanced residual block, the output of the twelfth convolutional layer of the first spatial angle convolutional layer outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in the first enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the first spatial angle convolution layer in the first enhanced residual block receives The reorganization operation result of all feature maps in the first enhanced residual block is that the output end of the thirteenth convolution layer of the first spatial angle convolution layer in the first enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the first spatial angle convolution layer in the first enhanced residual block. The set composed of these feature maps is recorded as The input end of the tenth convolutional layer in the second spatial feature transformation layer in the first enhanced residual block receives F Align,1 All feature maps in the first enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the second spatial feature transformation layer in the first enhanced residual block outputs 64 frames. And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the second spatial feature transformation layer in the first enhanced residual block receives F Align,1 All feature maps in the first enhanced residual block have 64 outputs at the output end of the eleventh convolutional layer in the second spatial feature transformation layer. And the height is The feature map of the output is recorded as The input end of the second spatial feature transformation layer in the first enhanced residual block receives All feature maps in All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the second spatial feature transformation layer in the first enhanced residual block. The set of these feature maps is recorded as The input end of the twelfth convolutional layer in the second spatial angle convolutional layer in the first enhanced residual block receives For all feature maps in the first enhanced residual block, the output of the twelfth convolutional layer of the second spatial angle convolutional layer outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in the first enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the second spatial angle convolution layer in the first enhanced residual block receives The reorganization operation result of all feature maps in the first enhanced residual block is that the output end of the thirteenth convolution layer of the second spatial angle convolution layer in the first enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the second space angle convolution layer in the first enhanced residual block. The set composed of these feature maps is recorded as The input of the global mean pooling layer in the channel attention layer in the first enhanced residual block receives All feature maps in the first enhanced residual block have a global mean pooling layer in the channel attention layer, and the output of the global mean pooling layer in the channel attention layer in the first enhanced residual block has a width of 64. And the height is The set of all the output feature maps is recorded as F GAP,1 , F GAP,1 All feature values in each feature map in are the same; the input end of the fourteenth convolutional layer in the channel attention layer in the first enhanced residual block receives F GAP,1 For all feature maps in the first enhanced residual block, the output of the fourteenth convolutional layer in the channel attention layer outputs 4 frames with a width of And the height is The set of all the output feature maps is recorded as F DS,1 ; The input of the fifteenth convolutional layer in the channel attention layer in the first enhanced residual block receives F DS,1 All feature maps in the first enhanced residual block have a width of 64, and the output of the fifteenth convolutional layer in the channel attention layer in the first enhanced residual block has a width of And the height is The set of all the output feature maps is recorded as F US,1 ; F US,1 All feature maps in All feature maps in are multiplied element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the channel attention layer in the first enhanced residual block. The set of these feature maps is recorded as F CA,1 ; F CA,1 All feature maps in F LR All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the first enhanced residual block. The set of these feature maps is F En,1 ; The input end of the tenth convolutional layer in the first spatial feature transformation layer in the second enhanced residual block receives F Align,2 All feature maps in the second enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the first spatial feature transformation layer in the second enhanced residual block outputs 64 frames. And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the first spatial feature transformation layer in the second enhanced residual block receives F Align,2 All feature maps in the second enhanced residual block output 64 images with a width of And the height is The feature map of the output is recorded as The receiving end of the first spatial feature transformation layer in the second enhanced residual block receives F En,1 All feature maps in F En,1 All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the first spatial feature transformation layer in the second enhanced residual block. The set of these feature maps is recorded as The input end of the twelfth convolutional layer in the first spatial angle convolutional layer in the second enhanced residual block receives For all feature maps in the second enhanced residual block, the output of the twelfth convolutional layer of the first spatial angle convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as right All feature maps in the second enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the first spatial angle convolution layer in the second enhanced residual block receives The reorganization operation result of all feature maps in the second enhanced residual block is that the output end of the thirteenth convolution layer of the first spatial angle convolution layer in the second enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the first spatial angle convolution layer in the second enhanced residual block. The set composed of these feature maps is recorded as The input end of the tenth convolutional layer in the second spatial feature transformation layer in the second enhanced residual block receives F Align,2 All feature maps in the second spatial feature transformation layer in the second enhanced residual block output 64 widths of the output end of the tenth convolutional layer And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the second spatial feature transformation layer in the second enhanced residual block receives F Align,2 All feature maps in the second spatial feature transformation layer in the second enhanced residual block output 64 widths of the eleventh convolutional layer. And the height is The feature map of the output is recorded as The receiving end receives the second spatial feature transformation layer in the second enhanced residual block All feature maps in All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the second spatial feature transformation layer in the second enhanced residual block. The set of these feature maps is recorded as The input end of the twelfth convolution layer in the second spatial angle convolution layer in the second enhanced residual block receives All feature maps in the second enhanced residual block have 64 pixels of width at the output of the twelfth convolution layer of the second spatial angle convolution layer. And the height is The feature map of the output is recorded as right All feature maps in the second enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the second spatial angle convolution layer in the second enhanced residual block receives The reorganization operation result of all feature maps in the second enhanced residual block is that the output end of the thirteenth convolution layer of the second spatial angle convolution layer in the second enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the second spatial angle convolution layer in the second enhanced residual block. The set composed of these feature maps is recorded as The input of the global mean pooling layer in the channel attention layer in the second enhanced residual block receives All feature maps in the second enhanced residual block have a global mean pooling layer in the channel attention layer, and the output of the global mean pooling layer in the channel attention layer in the second enhanced residual block has a width of 64. And the height is The set of all the output feature maps is recorded as F GAP,2 , F GAP,2 All feature values in each feature map in are the same; the input end of the fourteenth convolutional layer in the channel attention layer in the second enhanced residual block receives F GAP,2 All feature maps in the second enhanced residual block have 4 outputs at the output of the 14th convolutional layer in the channel attention layer. And the height is The set of all the output feature maps is recorded as F DS,2 ; The input end of the fifteenth convolutional layer in the channel attention layer in the second enhanced residual block receives F DS,2 All feature maps in the second enhanced residual block have a width of 64, and the output of the fifteenth convolutional layer in the channel attention layer in the second enhanced residual block has a width of And the height is The set of all the output feature maps is recorded as F US,2 ; F US,2 All feature maps in All feature maps in are multiplied element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the channel attention layer in the second enhanced residual block. The set of these feature maps is recorded as F CA,2 ; F CA,2 All feature maps in F En,1 All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the second enhanced residual block. The set of these feature maps is F En,2 ; The input end of the tenth convolutional layer in the first spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the first spatial feature transformation layer in the third enhanced residual block outputs 64 And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the first spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have 64 outputs at the output end of the eleventh convolutional layer in the first spatial feature transformation layer. And the height is The feature map of the output is recorded as The receiving end of the first spatial feature transformation layer in the third enhanced residual block receives F En,2 All feature maps in F En,2 All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the first spatial feature transformation layer in the third enhanced residual block. The set of these feature maps is recorded as The input end of the twelfth convolutional layer in the first spatial angle convolutional layer in the third enhanced residual block receives For all feature maps in the third enhanced residual block, the output of the twelfth convolutional layer of the first spatial angle convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as right All feature maps in the third enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the first spatial angle convolution layer in the third enhanced residual block receives The reorganization operation result of all feature maps in the third enhanced residual block is that the output end of the thirteenth convolution layer of the first spatial angle convolution layer in the third enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the first space angle convolution layer in the third enhanced residual block. The set composed of these feature maps is recorded as The input end of the tenth convolutional layer in the second spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have a width of 64, and the output of the tenth convolutional layer in the second spatial feature transformation layer in the third enhanced residual block outputs 64 And the height is The feature map of the output is recorded as The input end of the eleventh convolutional layer in the second spatial feature transformation layer in the third enhanced residual block receives F Align,3 All feature maps in the third enhanced residual block have 64 outputs with a width of And the height is The feature map of the output is recorded as The receiving end receives the second spatial feature transformation layer in the third enhanced residual block All feature maps in All feature maps in All feature maps in are multiplied element by element, and then the multiplication result is combined with All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the second spatial feature transformation layer in the third enhanced residual block. The set of these feature maps is recorded as The input end of the twelfth convolution layer in the second spatial angle convolution layer in the third enhanced residual block receives For all feature maps in the third enhanced residual block, the output of the twelfth convolutional layer of the second spatial angle convolutional layer outputs 64 images with a width of And the height is The feature map of the output is recorded as right All feature maps in the third enhanced residual block are reorganized from the spatial dimension to the angular dimension, and the input end of the thirteenth convolution layer in the second spatial angle convolution layer in the third enhanced residual block receives The reorganization operation result of all feature maps in the third enhanced residual block is that the output end of the thirteenth convolution layer of the second spatial angle convolution layer in the third enhanced residual block outputs 64 frames with a width of And the height is The feature map of the output is recorded as right All feature maps in are reorganized from the angle dimension to the space dimension, and all feature maps obtained after the reorganization operation are used as all feature maps output from the output end of the second spatial angle convolution layer in the third enhanced residual block. The set composed of these feature maps is recorded as The input of the global mean pooling layer in the channel attention layer in the third enhanced residual block receives All feature maps in the third enhanced residual block have a global mean pooling layer in the channel attention layer, and the output of the global mean pooling layer in the channel attention layer in the third enhanced residual block has a width of 64. And the height is The set of all the output feature maps is recorded as F GAP,3 , F GAP,3 All feature values in each feature map in are the same; the input end of the fourteenth convolutional layer in the channel attention layer in the third enhanced residual block receives F GAP,3 All feature maps in the third enhanced residual block have 4 outputs at the output of the fourteenth convolutional layer in the channel attention layer. And the height is The set of all the output feature maps is recorded as F DS,3 ; The input end of the fifteenth convolutional layer in the channel attention layer in the third enhanced residual block receives F DS,3 All feature maps in the third enhanced residual block have a width of 64, and the output of the fifteenth convolutional layer in the channel attention layer in the third enhanced residual block has a width of And the height is The set of all the output feature maps is recorded as F US,3 ; F US,3 All feature maps in All feature maps in are multiplied element by element, and all the obtained feature maps are used as all feature maps outputted from the output end of the channel attention layer in the third enhanced residual block. The set of these feature maps is recorded as F CA,3 ; F CA,3 All feature maps in F En,2 All feature maps in are added element by element, and all the obtained feature maps are used as all feature maps output at the output end of the third enhanced residual block. The set of these feature maps is F En,3 ; Among them, the size of the convolution kernel of the tenth convolution layer and the eleventh convolution layer in the first enhanced residual block, the second enhanced residual block and the third enhanced residual block is 3×3, the convolution step size is 1, the number of input channels is 64, the number of output channels is 64, and no activation function is used. The size of the convolution kernel of the twelfth convolution layer and the thirteenth convolution layer in the first enhanced residual block, the second enhanced residual block and the third enhanced residual block is 3×3, the convolution step size is 1, the number of input channels is 64, the number of output channels is 64, and the activation function used is All are "ReLU". The size of the convolution kernel of the fourteenth convolution layer in the first enhanced residual block, the second enhanced residual block, and the third enhanced residual block is 1×1, the convolution step is 1, the number of input channels is 64, the number of output channels is 4, and the activation function used is "ReLU". The size of the convolution kernel of the fifteenth convolution layer in the first enhanced residual block, the second enhanced residual block, and the third enhanced residual block is 1×1, the convolution step is 1, the number of input channels is 4, the number of output channels is 64, and the activation function used is "Sigmoid".
