Light field image space super-resolution method based on local and global complementary information

Abstract

The invention discloses a light field image spatial super-resolution method based on local and global complementary information, which relates to the technical field of image processing, and comprises the following steps: extracting an initial light field feature map from an input low-resolution light field image by using a convolutional layer; utilizing a local-global space angle complementary information fusion module to extract local-global space angle complementary information from the initial light field feature map to obtain depth features of the light field image; and obtaining a high-resolution light field image by using a convolutional layer and a pixel recombination operation based on the depth feature of the light field image. According to the method, the problems that the complementary relationship between different sub-aperture images and different macro-pixel images is ignored and the global complementary information in the light field image is not fully utilized in the existing method are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a light field image spatial super-resolution method based on local-global complementary information. Background Art

[0002] Light field images are an image format that can record three-dimensional scene information. They can simultaneously capture the intensity and direction of light at different positions in space, so that they can perceive the depth of the scene and perform viewpoint transformations. They have a wide range of applications. However, due to the special structure of light field cameras, the spatial resolution of light field images is sacrificed while obtaining multi-viewpoint images, which limits the application of light field images to a certain extent. Therefore, it is of great significance to study methods to increase the spatial resolution of light field images.

[0003] The current mainstream light field image super-resolution method is to use deep learning technology and neural networks to learn the mapping from low-resolution light field images to high-resolution light field images. Since light field images record images from different perspectives, the premise of achieving high-quality and high-resolution reconstruction is to make full use of complementary information from different angles to enhance the details of the image. However, most of the current methods adopt a "divide and conquer" strategy, that is, to extract the features of the light field image from different subspaces of the light field image, namely, sub-aperture images, macro-pixel images, and polar plane images. Although this method has been proven to be effective and efficient, it ignores the complementary relationship between different sub-aperture images and different macro-pixel images, and the global complementary information in the light field image is not fully utilized, which limits the performance of the algorithm. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a light field image spatial super-resolution method based on local-global complementary information, which solves the problem that the existing methods ignore the complementary relationship between different sub-aperture images and different macro-pixel images, and the insufficient utilization of global complementary information in light field images.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a light field image spatial super-resolution method based on local-global complementary information, comprising the following steps:

[0006] S1: Use the convolutional layer to extract the initial light field feature map from the input low-resolution light field image;

[0007] S2: Using the local-global spatial angle complementary information fusion module to extract the local-global spatial angle complementary information from the initial light field feature map, and obtain the depth feature of the light field image;

[0008] S3: Based on the deep features of light field images, high-resolution light field images are obtained using convolutional layers and pixel reconstruction operations.

[0009] Furthermore, the low-resolution light field image in S1 is:

[0010]

[0011] Among them, L lr is a low-resolution light field image, is the scope, U and V are the angular resolution, H and W are the spatial resolution;

[0012] The initial light field characteristic graph is:

[0013]

[0014] Among them, F init is the initial light field feature map, and C is the feature dimension.

[0015] Furthermore, the local-global spatial angle complementary information fusion module in S2 performs the following operations:

[0016] S21: Use the sub-aperture image internal feature extraction module to extract local spatial information from the initial light field feature map. The formula is:

[0017] F localSpa =h localSpa (F init )

[0018] Among them, F localSpa is the local spatial information, h localSpa (.) is the local space feature extraction module;

[0019] S22: Based on the local spatial information, the local angle information is extracted using the macro pixel image internal feature extraction module, and the formula is:

[0020] F localAng =h localAng (F localSpa )

[0021] Among them, F localAng is the local angle information, h localAng (.) is the local angle feature extraction module;

[0022] S23: Based on the local angle information, the first light field structure information is extracted using the first horizontal polar plane image feature extraction module and the first vertical polar plane image feature extraction module, and the formula is:

[0023]

[0024] in, is the first light field structure information, h epi-v(.) is the vertical feature extraction module, h epi-h (.) is the horizontal feature extraction module;

[0025] S24: Using the feature extraction module between sub-aperture images to extract global spatial information from the initial light field feature map, the formula is:

[0026] F globalSpa =h globalSpa (F)

[0027] Among them, F globalSpa is the global spatial information, h globalSpa (.) is the global space feature extraction module;

[0028] S25: Based on the global spatial information, the global angle information is extracted using the feature extraction module between macro-pixel images. The formula is:

[0029] F globalAng =h globalAng (F globalSpa )

[0030] Among them, F globalAng is the global angle information, h globalAng (.) is the global angle feature extraction module;

[0031] S26: Based on the global angle information, the second light field structure information is extracted using the second horizontal polar plane image feature extraction module and the second vertical polar plane image feature extraction module, and the formula is:

[0032]

[0033] in, is the second light field structure information;

[0034] S27: The first light field structure information and the second light field structure information are feature spliced ​​according to the number of channels, and feature fusion is performed using deep convolution. The formula is:

[0035]

[0036] Among them, F fuse is the fused feature, DWConv(.) is the depth convolution, and Concat(.) is the connection operation;

[0037] S28: Repeat the above steps S21-S27 four times to fully extract the local-global spatial angle complementary features of the light field image and obtain the depth features of the light field image. The formula is:

[0038]

[0039] Among them, Fdeep is the depth feature of the light field image, is the output of the fourth local-global spatial angle complementary information fusion module, is the output of the third local-global spatial angle complementary information fusion module, is the output of the second local-global spatial angle complementary information fusion module, It is the output of the first local-global spatial angle complementary information fusion module.

[0040] Furthermore, the sub-aperture image internal feature extraction module performs the following operations:

[0041] A1: Convert the initial light field feature map into a sub-aperture image sequence. The formula is:

[0042]

[0043] Among them, F SAI is the feature map converted in the sub-aperture image internal feature extraction module, and Reshape(.) is the feature reorganization module;

[0044] A2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0045]

[0046] in, is the feature map after embedding position information in the internal feature extraction module of the sub-aperture image, H PAPE (.) is the output of the pixel attention module;

[0047] A3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the local spatial information of the light field. The formula is:

[0048]

[0049] in, is the local spatial information of the light field in the sub-aperture image internal feature extraction module, is the output of the second use of the strip window attention mechanism, This is the output of the first use of the strip window attention mechanism;

[0050] A4: Restore the enhanced features to a 4-dimensional tensor structure. The formula is:

[0051]

[0052] Among them, F EN1 is the output of the internal feature extraction module of the sub-aperture image.

[0053] Furthermore, the macro-pixel image internal feature extraction module performs the following operations:

[0054] B1: Convert the light field feature map into a macro-pixel image sequence. The formula is:

[0055]

[0056] Among them, F MacPI It is the feature map converted in the internal feature extraction module of the macro pixel image;

[0057] B2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0058]

[0059] in, It is a feature map after embedding position information in the internal feature extraction module of the macro pixel image;

[0060] B3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the local angle information of the light field. The formula is:

[0061]

[0062] in, It is the local angle information of the light field in the internal feature extraction module of the macro-pixel image;

[0063] B4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is:

[0064]

[0065] Among them, F EN2 It is the output of the internal feature extraction module of the macro-pixel image.

[0066] Furthermore, the sub-aperture image feature extraction module performs the following operations:

[0067] C1: Convert the light field feature map into a sub-aperture image block array. The formula is:

[0068]

[0069] Among them, F SAIs is the feature map converted in the feature extraction module between sub-aperture images, h and w are the sizes of the sub-aperture image blocks;

[0070] C2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0071]

[0072] in, It is a feature map after embedding position information in the feature extraction module between sub-aperture images;

[0073] C3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the global spatial information of the light field. The formula is:

[0074]

[0075] in, is the global spatial information of the light field in the feature extraction module between sub-aperture images;

[0076] C4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is:

[0077]

[0078] Among them, F EN3 is the output of the feature extraction module between sub-aperture images.

[0079] Furthermore, the macro pixel image feature extraction module performs the following operations:

[0080] D1: Convert the light field feature map into a macro-pixel image array. The formula is:

[0081]

[0082] Among them, F MacPIs is the feature map converted in the feature extraction module between macro-pixel images, and n is the number of macro-pixel images;

[0083] D2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0084]

[0085] in, It is a feature map after embedding position information in the feature extraction module between macro-pixel images;

[0086] D3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the global angle information of the light field. The formula is:

[0087]

[0088] in, It is the global angle information of the light field in the feature extraction module between macro-pixel images;

[0089] D4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is:

[0090]

[0091] Among them, F EN4 is the output of the feature extraction module between macro-pixel images.

[0092] Furthermore, the first horizontal polar plane image feature extraction module and the second horizontal polar plane image feature extraction module perform the following operations:

[0093] E1: Convert the light field feature map into a horizontal polar plane image. The formula is:

[0094]

[0095] Among them, F epi-h is a feature map converted in the first horizontal polar plane image feature extraction module or the second horizontal polar plane image feature extraction module;

[0096] E2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0097]

[0098] in, A feature map after embedding position information in the first horizontal polar plane image feature extraction module or the second horizontal polar plane image feature extraction module;

[0099] E3: Based on the feature map after embedding the position information, the light field structure information is extracted using the strip window attention mechanism twice in sequence. The formula is:

[0100]

[0101] in, The light field structure information in the first horizontal polar plane image feature extraction module or the second horizontal polar plane image feature extraction module;

[0102] E4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is:

[0103]

[0104] Among them, F EN5 It is the output of the first horizontal pole plane image feature extraction module or the second horizontal pole plane image feature extraction module.

[0105] Furthermore, the first vertical polar plane image feature extraction module and the second vertical polar plane image feature extraction module perform the following operations:

[0106] F1: Convert the light field feature map into a vertical polar plane image. The formula is:

[0107]

[0108] Among them, F epi-v is a feature map converted in the first vertical polar plane image feature extraction module or the second vertical polar plane image feature extraction module;

[0109] F2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0110]

[0111] in, A feature map after embedding position information in the first vertical polar plane image feature extraction module or the second vertical polar plane image feature extraction module;

[0112] F3: Based on the feature map after embedding the position information, the light field structure information is extracted using the strip window attention mechanism twice in sequence. The formula is:

[0113]

[0114] in, The light field structure information in the first vertical pole plane image feature extraction module or the second vertical pole plane image feature extraction module;

[0115] F4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is:

[0116]

[0117] Among them, F EN6 It is the output of the first vertical pole plane image feature extraction module or the second vertical pole plane image feature extraction module.

[0118] Furthermore, the high-resolution light field image in S3 is:

[0119] L hr =bicubic(L lr )+Conv(pixelHhuffle(Conv(F deep )))

[0120] Among them, L hr is a high-resolution light field image, Conv(.) is a convolution operation, bicubic(.) is a bicubic interpolation function, and pixelShuffle(.) is a pixel reorganization module.

[0121] The beneficial effects of the present invention are as follows: the method of the present invention has clear modules and simple steps, can make full use of the local and global information of the light field image, can fully maintain the structural characteristics of the light field image, and can well restore a clear high-resolution light field image. In addition, we use a strip window attention mechanism in the invented model, which can better meet the efficient reconstruction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] Figure 1 This is a flow chart of the light field image spatial super-resolution method based on local-global complementary information of the present invention.

[0123] Figure 2 This is a framework diagram of the spatial super-resolution of light field images based on local-global complementary information of the present invention.

[0124] Figure 3 This is a 2x light field image super-resolution visualization.

[0125] Figure 4 This is a visualization of the 4x light field image super-resolution. DETAILED DESCRIPTION

[0126] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0127] Embodiment 1, as Figure 1 and Figure 2 As shown, a light field image spatial super-resolution method based on local-global complementary information includes the following steps:

[0128] S1: Use the convolutional layer to extract the initial light field feature map from the input low-resolution light field image;

[0129] S2: Using the local-global spatial angle complementary information fusion module to extract the local-global spatial angle complementary information from the initial light field feature map, and obtain the depth feature of the light field image;

[0130] S3: Based on the deep features of light field images, high-resolution light field images are obtained using convolutional layers and pixel reconstruction operations.

[0131] In one embodiment of the present invention, an initial convolution with a kernel size of 1×3×3 and three cascaded 1×3×3 convolutional networks are used to input a light field image L with a spatial resolution of H×W and an angular resolution of U×V. lr Perform feature extraction to obtain initial high-dimensional features.

[0132] The low-resolution light field image in S1 is:

[0133]

[0134] Among them, Llr is a low-resolution light field image, is the scope, U and V are the angular resolution, H and W are the spatial resolution;

[0135] The initial light field characteristic graph is:

[0136]

[0137] Among them, F init is the initial light field feature map, and C is the feature dimension.

[0138] The local-global spatial angle complementary information fusion module in S2 performs the following operations:

[0139] S21: Use the sub-aperture image internal feature extraction module to extract local spatial information from the initial light field feature map. The formula is:

[0140] F localSpa =h localSpa (F init )

[0141] Among them, F localSpa is the local spatial information, h localSpa (.) is the local space feature extraction module;

[0142] S22: Based on the local spatial information, the local angle information is extracted using the macro pixel image internal feature extraction module, and the formula is:

[0143] F localAng =h localAng (F localSpa )

[0144] Among them, F localAng is the local angle information, h localAng (.) is the local angle feature extraction module;

[0145] S23: Based on the local angle information, the first light field structure information is extracted using the first horizontal polar plane image feature extraction module and the first vertical polar plane image feature extraction module, and the formula is:

[0146]

[0147] in, is the first light field structure information, h epi-v (.) is the vertical feature extraction module, h epi-h (.) is the horizontal feature extraction module;

[0148] S24: Using the feature extraction module between sub-aperture images to extract global spatial information from the initial light field feature map, the formula is:

[0149] F globalSpa =h globalSpa (F)

[0150] Among them, F globalSpa is the global spatial information, h globalSpa (.) is the global space feature extraction module;

[0151] S25: Based on the global spatial information, the global angle information is extracted using the feature extraction module between macro-pixel images. The formula is:

[0152] F globalAng =h globalAng (F globalSpa )

[0153] Among them, F globalAng is the global angle information, h globalAng (.) is the global angle feature extraction module;

[0154] S26: Based on the global angle information, the second light field structure information is extracted using the second horizontal polar plane image feature extraction module and the second vertical polar plane image feature extraction module, and the formula is:

[0155]

[0156] in, is the second light field structure information;

[0157] S27: The first light field structure information and the second light field structure information are feature spliced ​​according to the number of channels, and feature fusion is performed using deep convolution. The formula is:

[0158]

[0159] Among them, F fuse is the fused feature, DWConv(.) is the depth convolution, and Concat(.) is the connection operation;

[0160] S28: Repeat the above steps S21-S27 four times to fully extract the local-global spatial angle complementary features of the light field image and obtain the depth features of the light field image. The formula is:

[0161]

[0162] Among them, F deep is the depth feature of the light field image, is the output of the fourth local-global spatial angle complementary information fusion module, is the output of the third local-global spatial angle complementary information fusion module, is the output of the second local-global spatial angle complementary information fusion module, It is the output of the first local-global spatial angle complementary information fusion module.

[0163] The sub-aperture image internal feature extraction module performs the following operations:

[0164] A1: Convert the initial light field feature map into a sub-aperture image sequence. The formula is:

[0165]

[0166] Among them, F SAI is the feature map converted in the sub-aperture image internal feature extraction module, and Reshape(.) is the feature reorganization module;

[0167] A2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0168]

[0169] in, is the feature map after embedding position information in the internal feature extraction module of the sub-aperture image, H PAPE (.) is the output of the pixel attention module;

[0170] A3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the local spatial information of the light field. The formula is:

[0171]

[0172] in, is the local spatial information of the light field in the sub-aperture image internal feature extraction module, is the output of the second use of the strip window attention mechanism, This is the output of the first use of the strip window attention mechanism;

[0173] A4: Restore the enhanced features to a 4-dimensional tensor structure. The formula is:

[0174]

[0175] Among them, F EN1 is the output of the internal feature extraction module of the sub-aperture image.

[0176] The pixel attention module H in this embodiment PAPE(·) includes the following steps: the input feature map is passed through a 3×3 deep convolutional network to obtain an attention map, i.e., map = DWConv(F); the above attention map is multiplied with the input feature map to obtain a feature map with embedded position information, i.e., where ⊙ is the Hadamard product.

[0177] The strip window attention mechanism H in this embodiment SWA (·) Includes the following steps:

[0178] For an input feature map First, the feature map is divided into two parts according to the channel, namely

[0179] For the two features obtained above, strip window division is performed respectively. Specifically: h , dividing it into horizontal strip windows, in Similarly, in

[0180] For all the above bar area feature maps f h ,f v Perform self-attention calculations respectively to obtain enhanced features. The self-attention calculation steps are:

[0181] Through the projection matrix W q ,W k ,W v Calculate q, k, v. q = W q f,k=W k f,v=W v f;

[0182] Calculating self-attention Where d is the feature dimension;

[0183] The above enhanced feature F h ,F v Merge by channel into a complete feature map F;

[0184] The above features are further passed through a linear layer-depth convolution-linear layer to obtain the final features.

[0185] The macro pixel image internal feature extraction module performs the following operations:

[0186] B1: Convert the light field feature map into a macro-pixel image sequence. The formula is:

[0187]

[0188] Among them, FMacPI It is the feature map converted in the internal feature extraction module of the macro pixel image;

[0189] B2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0190]

[0191] in, It is a feature map after embedding position information in the internal feature extraction module of the macro pixel image;

[0192] B3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the local angle information of the light field. The formula is:

[0193]

[0194] in, It is the local angle information of the light field in the internal feature extraction module of the macro-pixel image;

[0195] B4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is:

[0196]

[0197] Among them, F EN2 It is the output of the internal feature extraction module of the macro-pixel image.

[0198] The sub-aperture image feature extraction module performs the following operations:

[0199] C1: Convert the light field feature map into a sub-aperture image block array. The formula is:

[0200]

[0201] Among them, F SAIs is the feature map converted in the feature extraction module between sub-aperture images, h and w are the sizes of the sub-aperture image blocks;

[0202] C2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0203]

[0204] in, It is a feature map after embedding position information in the feature extraction module between sub-aperture images;

[0205] C3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the global spatial information of the light field. The formula is:

[0206]

[0207] in, is the global spatial information of the light field in the feature extraction module between sub-aperture images;

[0208] C4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is:

[0209]

[0210] Among them, F EN3 is the output of the feature extraction module between sub-aperture images.

[0211] The macro pixel image feature extraction module performs the following operations:

[0212] D1: Convert the light field feature map into a macro-pixel image array. The formula is:

[0213]

[0214] Among them, F MacPIs is the feature map converted in the feature extraction module between macro-pixel images, and n is the number of macro-pixel images;

[0215] D2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0216]

[0217] in, It is a feature map after embedding position information in the feature extraction module between macro-pixel images;

[0218] D3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the global angle information of the light field. The formula is:

[0219]

[0220] in, It is the global angle information of the light field in the feature extraction module between macro-pixel images;

[0221] D4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is:

[0222]

[0223] Among them, F EN4 is the output of the feature extraction module between macro-pixel images.

[0224] The first horizontal polar plane image feature extraction module and the second horizontal polar plane image feature extraction module perform the following operations:

[0225] E1: Convert the light field feature map into a horizontal polar plane image. The formula is:

[0226]

[0227] Among them, F epi-h is a feature map converted in the first horizontal polar plane image feature extraction module or the second horizontal polar plane image feature extraction module;

[0228] E2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0229]

[0230] in, A feature map after embedding position information in the first horizontal polar plane image feature extraction module or the second horizontal polar plane image feature extraction module;

[0231] E3: Based on the feature map after embedding the position information, the light field structure information is extracted using the strip window attention mechanism twice in sequence. The formula is:

[0232]

[0233] in, The light field structure information in the first horizontal polar plane image feature extraction module or the second horizontal polar plane image feature extraction module;

[0234] E4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is:

[0235]

[0236] Among them, F EN5 It is the output of the first horizontal pole plane image feature extraction module or the second horizontal pole plane image feature extraction module.

[0237] The first vertical polar plane image feature extraction module and the second vertical polar plane image feature extraction module perform the following operations:

[0238] F1: Convert the light field feature map into a vertical polar plane image. The formula is:

[0239]

[0240] Among them, F epi-v is a feature map converted in the first vertical polar plane image feature extraction module or the second vertical polar plane image feature extraction module;

[0241] F2: Use the pixel attention module to embed the position information of the converted feature map. The formula is:

[0242]

[0243] in, A feature map after embedding position information in the first vertical polar plane image feature extraction module or the second vertical polar plane image feature extraction module;

[0244] F3: Based on the feature map after embedding the position information, the light field structure information is extracted using the strip window attention mechanism twice in sequence. The formula is:

[0245]

[0246] in, The light field structure information in the first vertical pole plane image feature extraction module or the second vertical pole plane image feature extraction module;

[0247] F4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is:

[0248]

[0249] Among them, F EN6 It is the output of the first vertical pole plane image feature extraction module or the second vertical pole plane image feature extraction module.

[0250] The high-resolution light field image in S3 is:

[0251] L hr =bicubic(L kr )+Conv(pixelSHuffle(Conv(F deep )))

[0252] Among them, L hr is a high-resolution light field image, Conv(.) is a convolution operation, bicubic(.) is a bicubic interpolation function, and pixelShuffle(.) is a pixel reorganization module.

[0253] In one embodiment of the present invention, five open source light field image data sets, EPFL, HCInew, HCIold, INRIA, and STFgantry, are used for experiments. Among them, the EPFL and INRIA data sets are real data sets taken by the Lytro plenoptic camera, and the STFgantry data set is a real scene data set taken by the gantry camera. The HCIold and HCInew data sets are virtual light field data sets synthesized using simulation software. The present invention uses a bicubic difference method to downsample high-resolution images by 2 times and 4 times, respectively, as neural network input. The present invention divides the above five data sets into 144 scenes as training sets and 23 scenes as test sets.

[0254] The present invention uses peak signal-to-noise ratio (PSNR) and structural similarity metric (SSIM) as super-resolution evaluation indicators.

[0255] The results of this method on five datasets for 2x and 4x light field super-resolution tasks are shown in the following table.

[0256] Table 1 Light field image super-resolution test results

[0257] magnification EPFL HCInew HCIold INRIA STFgantry ×2 35.96 / .9828 38.36 / .9810 45.07 / .9950 37.71 / .9879 41.43 / .9952 ×4 29.78 / .9243 31.66 / .9248 37.95 / .9750 31.87 / .9546 32.22 / .9578

[0258] In addition to the quantitative evaluation, this example also makes a qualitative visual comparison with the best current super-resolution method by 2x and 4x. The results are as follows: Figure 3 and Figure 4 As shown, this shows that the scheme has a good effect.

[0259] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the invention.

Claims

1. A light field image spatial super-resolution method based on local-global complementary information, characterized in that: The following steps are involved: S1: Use the convolutional layer to extract the initial light field feature map from the input low-resolution light field image; S2: Using the local-global spatial angle complementary information fusion module to extract the local-global spatial angle complementary information from the initial light field feature map, and obtain the depth feature of the light field image; S3: Based on the deep features of light field images, high-resolution light field images are obtained using convolutional layers and pixel reconstruction operations.

2. The method for spatial super-resolution of light field images based on local-global complementary information according to claim 1, characterized in that: The low-resolution light field image in S1 is: Among them, L lr is a low-resolution light field image, is the scope, U and V are the angular resolution, H and W are the spatial resolution; The initial light field characteristic graph is: Among them, F init is the initial light field feature map, and C is the feature dimension.

3. The method for spatial super-resolution of light field images based on local-global complementary information according to claim 2, characterized in that: The local-global spatial angle complementary information fusion module in S2 performs the following operations: S21: Use the sub-aperture image internal feature extraction module to extract local spatial information from the initial light field feature map. The formula is: F localSpa =h localSpa (F init ) Among them, F localSpa is the local spatial information, h localSpa (.) is the local space feature extraction module; S22: Based on the local spatial information, the local angle information is extracted using the macro pixel image internal feature extraction module, and the formula is: F localAng =h licalAng (F licalSpa ) Among them, F localAng is the local angle information, h localAng (.) is the local angle feature extraction module; S23: Based on the local angle information, the first light field structure information is extracted using the first horizontal polar plane image feature extraction module and the first vertical polar plane image feature extraction module, and the formula is: in, is the first light field structure information, h epi-v (.) is the vertical feature extraction module, h epi-h (.) is the horizontal feature extraction module; S24: Using the feature extraction module between sub-aperture images to extract global spatial information from the initial light field feature map, the formula is: F globalSpa =h globalSpa (F) Among them, F globalSpa is the global spatial information, h globalSpa (.) is the global space feature extraction module; S25: Based on the global spatial information, the global angle information is extracted using the feature extraction module between macro-pixel images. The formula is: F globalAng =h globalAng (F globalSpa ) Among them, F globalang is the global angle information, h globalang (.) is the global angle feature extraction module; S26: Based on the global angle information, the second light field structure information is extracted using the second horizontal polar plane image feature extraction module and the second vertical polar plane image feature extraction module, and the formula is: in, is the second light field structure information; S27: The first light field structure information and the second light field structure information are feature spliced ​​according to the number of channels, and feature fusion is performed using deep convolution. The formula is: Among them, F fuse is the fused feature, DWConv(.) is the depth convolution, and DWCat(.) is the connection operation; S28: Repeat the above steps S21-S27 four times to fully extract the local-global spatial angle complementary features of the light field image and obtain the depth features of the light field image. The formula is: Among them, F deep is the depth feature of the light field image, is the output of the fourth local-global spatial angle complementary information fusion module, is the output of the third local-global spatial angle complementary information fusion module, is the output of the second local-global spatial angle complementary information fusion module, It is the output of the first local-global spatial angle complementary information fusion module.

4. The method for spatial super-resolution of light field images based on local-global complementary information according to claim 3, characterized in that: The sub-aperture image internal feature extraction module performs the following operations: A1: Convert the initial light field feature map into a sub-aperture image sequence. The formula is: Among them, F SAI is the feature map converted in the sub-aperture image internal feature extraction module, and Reshape(.) is the feature reorganization module; A2: Use the pixel attention module to embed the position information of the converted feature map. The formula is: in, is the feature map after embedding position information in the internal feature extraction module of the sub-aperture image, H PAPE (.) is the output of the pixel attention module; A3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the local spatial information of the light field. The formula is: in, is the local spatial information of the light field in the sub-aperture image internal feature extraction module, is the output of the second use of the strip window attention mechanism, This is the output of the first use of the strip window attention mechanism; A4: Restore the enhanced features to a 4-dimensional tensor structure. The formula is: Among them, F EN1 is the output of the internal feature extraction module of the sub-aperture image.

5. The method for spatial super-resolution of light field images based on local-global complementary information according to claim 4, characterized in that: The macro pixel image internal feature extraction module performs the following operations: B1: Convert the light field feature map into a macro-pixel image sequence. The formula is: Among them, F MacPI It is the feature map converted in the internal feature extraction module of the macro pixel image; B2: Use the pixel attention module to embed the position information of the converted feature map. The formula is: in, It is a feature map after embedding position information in the internal feature extraction module of the macro pixel image; B3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the local angle information of the light field. The formula is: in, It is the local angle information of the light field in the internal feature extraction module of the macro-pixel image; B4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is: Among them, F EN2 It is the output of the internal feature extraction module of the macro-pixel image.

6. The method for spatial super-resolution of light field images based on local-global complementary information according to claim 5, characterized in that: The sub-aperture image feature extraction module performs the following operations: C1: Convert the light field feature map into a sub-aperture image block array. The formula is: Among them, F SAIs is the feature map converted in the feature extraction module between sub-aperture images, h and w are the sizes of the sub-aperture image blocks; C2: Use the pixel attention module to embed the position information of the converted feature map. The formula is: in, It is a feature map after embedding position information in the feature extraction module between sub-aperture images; C3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the global spatial information of the light field. The formula is: in, is the global spatial information of the light field in the feature extraction module between sub-aperture images; C4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is: Among them, F EN3 is the output of the feature extraction module between sub-aperture images.

7. The method for spatial super-resolution of light field images based on local-global complementary information according to claim 6, characterized in that: The macro pixel image feature extraction module performs the following operations: D1: Convert the light field feature map into a macro-pixel image array. The formula is: Among them, F MacPIs is the feature map converted in the feature extraction module between macro-pixel images, and n is the number of macro-pixel images; D2: Use the pixel attention module to embed the position information of the converted feature map. The formula is: in, It is a feature map after embedding position information in the feature extraction module between macro-pixel images; D3: Based on the feature map after embedding the position information, the strip window attention mechanism is used twice in sequence to extract the global angle information of the light field. The formula is: in, It is the global angle information of the light field in the feature extraction module between macro-pixel images; D4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is: Among them, F EN4 is the output of the feature extraction module between macro-pixel images.

8. The method for spatial super-resolution of light field images based on local-global complementary information according to claim 7, characterized in that: The first horizontal polar plane image feature extraction module and the second horizontal polar plane image feature extraction module perform the following operations: E1: Convert the light field feature map into a horizontal polar plane image. The formula is: Among them, F epi-h is a feature map converted in the first horizontal polar plane image feature extraction module or the second horizontal polar plane image feature extraction module; E2: Use the pixel attention module to embed the position information of the converted feature map. The formula is: in, A feature map after embedding position information in the first horizontal polar plane image feature extraction module or the second horizontal polar plane image feature extraction module; E3: Based on the feature map after embedding the position information, the light field structure information is extracted using the strip window attention mechanism twice in sequence. The formula is: in, The light field structure information in the first horizontal polar plane image feature extraction module or the second horizontal polar plane image feature extraction module; E4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is: Among them, F EN5 It is the output of the first horizontal pole plane image feature extraction module or the second horizontal pole plane image feature extraction module.

9. The method for spatial super-resolution of light field images based on local-global complementary information according to claim 8, characterized in that: The first vertical polar plane image feature extraction module and the second vertical polar plane image feature extraction module perform the following operations: F1: Convert the light field feature map into a vertical polar plane image. The formula is: Among them, F epi-v is a feature map converted in the first vertical polar plane image feature extraction module or the second vertical polar plane image feature extraction module; F2: Use the pixel attention module to embed the position information of the converted feature map. The formula is: in, A feature map after embedding position information in the first vertical polar plane image feature extraction module or the second vertical polar plane image feature extraction module; F3: Based on the feature map after embedding the position information, the light field structure information is extracted using the strip window attention mechanism twice in sequence. The formula is: in, The light field structure information in the first vertical pole plane image feature extraction module or the second vertical pole plane image feature extraction module; F4: Restore the enhanced features into a 4-dimensional tensor structure, the formula is: Among them, F EN6 It is the output of the first vertical pole plane image feature extraction module or the second vertical pole plane image feature extraction module.

10. The method for spatial super-resolution of light field images based on local-global complementary information according to claim 3, characterized in that: The high-resolution light field image in S3 is: L hr =bicubic(L lr )+Conv(pixelHhuffle(Conv(F deep ))) Among them, L hr is a high-resolution light field image, Conv(.) is a convolution operation, bicubic(.) is a bicubic interpolation function, and pixelShuffle(.) is a pixel reorganization module.

Images