Light Field Image Super-Resolution Reconstruction Method Combining Spatial and Angular Features

Abstract

The present invention specifically relates to a light field image super-resolution reconstruction method that fuses spatial and angular features, including: inputting a target light field image into a trained super-resolution reconstruction model to output a super-resolution reconstructed image; during model training: extracting the 2D spatial features and 2D angular features of each sub-aperture image in the low-resolution light field image, and then connecting the 2D spatial features and 2D angular features to generate global features; performing fusion enhancement on the global features to generate 4D light field features of the light field image; performing super-resolution reconstruction based on the 4D light field features of the light field image to generate a super-resolution reconstructed image; calculating the training loss according to the super-resolution reconstructed image and the high-resolution light field image used as a sample, and optimizing the parameters of the super-resolution reconstruction model until the model converges. The present invention can fully consider the 2D spatial features and 2D angular features of the light field image, and can effectively complement the spatial feature information and angular feature information in the light field image.

Description

Technical Field

[0001] The present invention relates to the technical field of light field image super-resolution reconstruction, and particularly relates to a light field image super-resolution reconstruction method that fuses spatial and angular features. Background Art

[0002] At present, the light field camera based on a microlens array, as a new generation of imaging system, has been widely applied in fields such as depth estimation, three-dimensional reconstruction, object segmentation, and image refocusing. Due to the spatial limitation of the microlens array, the spatial resolution of the sub-aperture images of the light field image is much lower than the resolution of the two-dimensional images acquired by traditional imaging devices. Therefore, how to improve the spatial resolution of the light field image has become a research hotspot in related fields, and the light field image super-resolution technology can better make up for the deficiency of its spatial resolution.

[0003] The existing light field image super-resolution reconstruction methods are mainly divided into traditional super-resolution methods and deep learning-based super-resolution methods. For the traditional light field image super-resolution reconstruction methods, they mainly utilize the prior assumptions of the actual shooting scene to propose corresponding physical models for implementation. However, limited by the calculation accuracy of the spatial domain scene prior, it is difficult for the traditional super-resolution methods to handle problems such as occlusion, noise, and image blur. The deep learning-based light field image super-resolution reconstruction methods implicitly construct the mapping relationship from the low-resolution image (LR) to the high-resolution image (HR), and do not require explicit calculation of the scene prior. Therefore, the deep learning-based light field image super-resolution reconstruction methods have better performance.

[0004] The applicant found that due to factors such as edge occlusion, the pixel distribution of each view image in the light field is uneven. However, the texture detail features of a distorted view can be restored and enhanced with the help of the supplementary information of other angular view images, that is, the angular feature information can effectively learn the redundant information in the light field image. However, most of the existing deep learning-based super-resolution methods only utilize the two-dimensional (2D) spatial feature information in the light field image for processing, and do not fully consider the angular feature information of the light field sub-aperture image array, resulting in poor results in the light field image super-resolution reconstruction. Therefore, how to design a light field image super-resolution reconstruction method that can fully consider the spatial feature information and angular feature information of the light field image is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In view of the deficiencies of the above-mentioned existing technologies, the technical problem to be solved by the present invention is: how to provide a light field image super-resolution reconstruction method that fuses spatial and angular features, can fully consider the 2D spatial features and 2D angular features of the light field image, and can achieve effective complementarity between the spatial feature information and angular feature information in the light field image, thereby improving the effectiveness and accuracy of the light field image super-resolution reconstruction.

[0006] To solve the above technical problem, the present invention adopts the following technical solutions:

[0007] A light field image super-resolution reconstruction method that fuses spatial and angular features, comprising:

[0008] S1: Obtain the target light field image to be reconstructed;

[0009] S2: Input the target light field image into the trained super-resolution reconstruction model, and output the corresponding super-resolution reconstructed image;

[0010] The super-resolution reconstruction model is trained through the following steps:

[0011] S201: Convert the high-resolution light field image used as a sample into a low-resolution light field image and use it as the model input;

[0012] S202: Extract the 2D spatial features of each sub-aperture image in the low-resolution light field image;

[0013] S203: Extract the 2D angular features of each sub-aperture image on the basis of the 2D spatial features, and then connect the 2D spatial features and 2D angular features of each sub-aperture image to generate the corresponding global features;

[0014] S204: Fuse and enhance the global features of each sub-aperture image to generate the 4D light field features of the light field image;

[0015] S205: Perform super-resolution reconstruction according to the 4D light field features of the light field image to generate the corresponding super-resolution reconstructed image;

[0016] S206: Calculate the training loss according to the super-resolution reconstructed image and the high-resolution light field image used as a sample, and optimize the parameters of the super-resolution reconstruction model based on the training loss;

[0017] S207: Repeat steps S201 to S206 until the super-resolution reconstruction model converges;

[0018] S3: Use the output super-resolution reconstructed image as the super-resolution reconstruction result of the target light field image.

[0019] Preferably, bicubic downsampling is performed on the high-resolution light field image used as a sample to obtain a low-resolution light field image.

[0020] Preferably, the low-resolution light field image is used as the input of the spatial feature extraction module, and the corresponding 2D spatial features are extracted through the spatial feature extraction module; wherein, the spatial feature extraction module includes a convolutional layer, a first ResBlock module, a first ResRFB module, a second ResBlock module, and a second ResRFB module that are connected end to end in sequence;

[0021] The convolutional layer is used to extract the shallow features of the low-resolution light field image;

[0022] The first ResBlock module and the second ResBlock module are used to activate the extracted features;

[0023] The first ResRFB module and the second ResRFB module are used to expand the receptive field through dilated convolution, strengthen the correlation between channel pixels, and then extract the corresponding spatial features; the output of the second ResRFB module is the final 2D spatial feature.

[0024] Preferably, both the first ResBlock module and the second ResBlock module include a 3×3 convolutional layer, a Relu activation function layer, and a 3×3 convolutional layer that are connected end to end in sequence.

[0025] Preferably, both the first ResRFB module and the second ResRFB module include a first 1×1 convolutional layer, three branches with their input ends respectively connected to the output end of the first 1×1 convolutional layer, a connection layer with its input ends respectively connected to the output ends of the three branches and used to connect the output features of the three branches, and a second 1×1 convolutional layer with its input end connected to the output end of the connection layer; a Relu activation function layer is arranged between the connection layer and the second 1×1 convolutional layer;

[0026] The three branches include: a first branch composed of a third 1×1 convolutional layer and a dilated convolutional layer with a dilation rate of 1, a second branch composed of a 3×3 convolutional layer and a dilated convolutional layer with a dilation rate of 3, and a third branch composed of a 5×5 convolutional layer and a dilated convolutional layer with a dilation rate of 5.

[0027] Preferably, the 2D spatial features of each sub-aperture image are used as the stack input, the 2D angular features of each sub-aperture image are extracted through the angular feature extraction module, and the 2D angular features and 2D spatial features of each sub-aperture image are connected to generate global features;

[0028] The angle feature extraction module includes several parallel branches for extracting new viewpoint features between 2D spatial feature pairs of different sub-aperture images, a 3×3 convolutional layer with its input end connected to the output ends of several parallel branches and used for merging the output features of each parallel branch, and a feature connection layer for connecting 2D angle features and 2D spatial features and generating global features;

[0029] The parallel branches include a 9×9 convolutional layer, a Relu activation function layer, a 5×5 convolutional layer, a Relu activation function layer, a 3×3 convolutional layer, and a Relu activation function layer that are connected end to end in sequence.

[0030] Preferably, the global feature of the sub-aperture image is represented by the following formula:

[0031]

[0032] In the formula: F G represents the global feature; F S represents the 2D spatial feature; F A represents the 2D angle feature; Convert G (·) represents a 2D convolution operation; concat(·) represents a feature connection operation; R represents a size transformation.

[0033] Preferably, the global features of each sub-aperture image are used as the input of the feature fusion and enhancement module, and after the global features are fused and enhanced by the feature fusion and enhancement module, 4D light field features are generated;

[0034] The feature fusion and enhancement module includes several spatial angle convolutional layers connected end to end in sequence. The spatial angle convolutional layer includes a spatial convolutional layer, a Relu activation function layer, an angle convolutional layer, and a Relu activation function layer connected end to end in sequence.

[0035] Preferably, the 4D light field features are used as the input of the reconstruction module, and the 4D light field features are super-resolution reconstructed by the reconstruction module based on the sub-pixel convolution upsampling method;

[0036] The reconstruction module first performs a 1×1 convolution on the 4D light field feature map with the number of channels C to expand its number of channels to a2×C; then performs pixel recombination to generate a high-resolution feature map with a resolution a times higher; finally, performs feature fusion on the high-resolution feature map through a convolutional layer to generate an array of super-resolved light field sub-aperture images, that is, a super-resolution reconstructed image.

[0037] Preferably, the training loss is calculated by the following formula:

[0038] loss=||L SR (u,v,s,t)-L HR (u,v,s,t)||1 ;

[0039] Where: L SR (u, v, s, t) represents the generated super-resolution reconstructed image; L HR (u, v, s, t) represents the high-resolution light field image used as a sample; (u, v) and (s, t) respectively represent the angular plane coordinates and spatial plane coordinates of the light field image.

[0040] Compared with the prior art, the light field image super-resolution reconstruction method that fuses spatial and angular features in the present invention has the following beneficial effects:

[0041] In the present invention, a super-resolution reconstruction model after training is used to perform super-resolution reconstruction on the light field image. The model first extracts the 2D spatial features of each sub-aperture image (i.e., view) in the light field image, then extracts the 2D angular features of each sub-aperture image based on the 2D spatial features, and then connects the 2D spatial features and 2D angular features to generate global features, and fuses and enhances them to obtain 4D light field features. Finally, super-resolution reconstruction is performed based on the 4D light field features to obtain the super-resolution reconstructed image. In the present invention, the 2D angular features of the light field image are further extracted based on the 2D spatial features, so that the spatial feature information and angular feature information of the sub-aperture image can be fully considered. On the one hand, the texture information of the light field is contained in different sub-aperture images, so that the spatial texture features play a key role in reconstructing the texture details of the high-resolution image. On the other hand, due to factors such as edge occlusion, the pixel distribution of each sub-aperture image in the light field is uneven, but the texture detail features of a distorted view can be restored and enhanced with the help of the supplementary information of other angular views, that is, the angular feature information can effectively learn the redundant information in the light field image and improve the geometric consistency of the super-resolution result. Therefore, the present invention can truly reconstruct the texture details in the light field image through the spatial feature information and angular feature information between different views (i.e., sub-aperture images) of the light field image, thereby improving the effectiveness of the super-resolution reconstruction of the light field image.

[0042] In the present invention, global features are generated by connecting 2D spatial features and 2D angular features, and the global features are fused and enhanced to generate 4D light field features. Then, super-resolution reconstruction is performed according to the 4D light field features to generate the super-resolution reconstructed image, so that the 2D spatial features and 2D angular features of each sub-aperture image in the light field image can be effectively fused, and thus the effective complementarity of the spatial feature information and angular feature information in the light field image can be realized, thereby further improving the accuracy of the super-resolution reconstruction of the light field image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the objectives, technical solutions, and advantages of the invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings, where:

[0044] Figure 1 It is a logic block diagram of a light field image super-resolution reconstruction method;

[0045] Figure 2 It is a schematic diagram of the overall architecture of the super-resolution reconstruction model;

[0046] Figure 3 It is a schematic diagram of the architecture of the spatial feature extraction module;

[0047] Figure 4 It is a schematic diagram of the architecture of the angular feature extraction module (FusionA);

[0048] Figure 5 It is a schematic diagram of the architecture of the feature fusion and enhancement module (FusionB);

[0049] Figure 6 They are the reconstruction results of origami by different methods;

[0050] Figure 7 They are the reconstruction results of bikes by different methods;

[0051] Figure 8 They are the reconstruction results of occlusions by different methods. Detailed implementation manners

[0052] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Components of the embodiments of the present invention usually described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.

[0053] It should be noted that similar reference numerals and letters denote similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship in which the inventive product is customarily placed during use. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance. In addition, terms such as "horizontal" and "vertical" do not mean that the components are required to be absolutely horizontal or hanging, but can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined. In the description of the present invention, it should also be noted that unless otherwise clearly specified and defined, the terms "set", "install", "connect", "couple" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0054] The following is a more detailed description through specific embodiments:

[0055] Embodiment:

[0056] This embodiment discloses a light field image super-resolution reconstruction method that fuses spatial and angular features.

[0057] As Figure 1 shown, the light field image super-resolution reconstruction method that fuses spatial and angular features includes:

[0058] S1: Obtain the target light field image to be reconstructed;

[0059] There are three representation methods for light field images, SAIs (sub-aperture images, sub-aperture image arrays), MacPI images (macro-pixel images), and EPI polar plane images. The present invention pre-decodes the light field image into the corresponding sub-aperture image array.

[0060] S2: Input the target light field image into the trained super-resolution reconstruction model and output the corresponding super-resolution reconstructed image;

[0061] Combined Figure 2 As shown, the super-resolution reconstruction model is trained through the following steps:

[0062] S201: Convert the high-resolution light field image used as a sample into a low-resolution light field image and use it as the model input;

[0063] S202: Extract the 2D spatial features of each sub-aperture image in the low-resolution light field image;

[0064] S203: Based on the 2D spatial features, extract the 2D angular features of each sub-aperture image, and then connect the 2D spatial features and 2D angular features of each sub-aperture image to generate the corresponding global features;

[0065] S204: Fuse and enhance the global features of each sub-aperture image to generate the 4D light field features of the light field image;

[0066] In this embodiment, the 2D spatial features are represented as [u×v:c:h:w]; the 2D angular features are represented as [h×w:c:u:v]; the 4D light field features are represented as [u:v:c:h:w].

[0067] S205: Perform super-resolution reconstruction based on the 4D light field features of the light field image to generate the corresponding super-resolution reconstruction image (i.e., the high-resolution light field sub-aperture image array);

[0068] S206: Calculate the training loss based on the super-resolution reconstruction image and the high-resolution light field image used as a sample, and optimize the parameters of the super-resolution reconstruction model based on the training loss;

[0069] S207: Repeat steps S201 to S206 until the super-resolution reconstruction model converges;

[0070] S3: Use the output super-resolution reconstruction image as the super-resolution reconstruction result of the target light field image.

[0071] In this embodiment, the so-called super-resolution reconstruction refers to improving the resolution of the original image through hardware or software methods, and obtaining a high-resolution image through a series of low-resolution images. The core idea of super-resolution reconstruction is to exchange time bandwidth (acquiring multiple frame image sequences of the same scene) for spatial resolution, and realize the conversion from time resolution to spatial resolution.

[0072] The present invention performs super-resolution reconstruction on a light field image through a trained super-resolution reconstruction model. The model first extracts the 2D spatial features of each sub-aperture image (i.e., view) in the light field image, then extracts the 2D angular features of each sub-aperture image based on the 2D spatial features, and then connects the 2D spatial features and the 2D angular features to generate global features, and fuses and enhances them to obtain 4D light field features. Finally, super-resolution reconstruction is performed based on the 4D light field features to obtain a super-resolution reconstructed image. The present invention further extracts the 2D angular features of the light field image based on the 2D spatial features, enabling full consideration of the spatial feature information and angular feature information of the sub-aperture images. On the one hand, the texture information of the light field is contained in different sub-aperture images, making the spatial texture features play a key role in reconstructing the texture details of the high-resolution image. On the other hand, due to factors such as edge occlusion, the pixel distribution of each sub-aperture image in the light field is uneven, but the texture detail features of a distorted view can be restored and enhanced with the supplementary information of other angular views, that is, the angular feature information can effectively learn the redundant information in the light field image and improve the geometric consistency of the super-resolution result. Therefore, the present invention can truly reconstruct the texture details in the light field image through the spatial feature information and angular feature information between different views (i.e., sub-aperture images) of the light field image, thereby improving the effectiveness of the super-resolution reconstruction of the light field image.

[0073] The present invention generates global features by connecting 2D spatial features and 2D angular features, fuses and enhances the global features to generate 4D light field features, and then performs super-resolution reconstruction according to the 4D light field features to generate a super-resolution reconstructed image, enabling effective fusion of the 2D spatial features and 2D angular features of each sub-aperture image in the light field image, and further enabling effective complementarity of the spatial feature information and angular feature information in the light field image, thereby further improving the accuracy of the super-resolution reconstruction of the light field image.

[0074] In terms of the overall architecture of the super-resolution reconstruction model, the present invention adopts an end-to-end network design, which can super-resolve the sub-aperture images of all viewpoints of the light field image at the same time, effectively reducing the number of network parameters and improving the super-resolution efficiency. Experimental results on 4 public databases show that compared with 5 existing methods, the super-resolution reconstruction model of the method of the present invention shows great advantages in both subjective vision and objective performance, can quickly and efficiently extract the contour texture detail features in the light field image, while effectively improving the spatial resolution of the light field image, using fewer network parameters, having high computational efficiency, and being able to super-resolve all sub-aperture image arrays at the same time.

[0075] In the specific implementation process, the high-resolution light field image used as a sample is subjected to bicubic downsampling (bicubic interpolation (Bicubic) downsampling) to obtain the corresponding low-resolution light field image.

[0076] When training the super-resolution reconstruction model of the present invention, bicubic downsampling is performed on the high-resolution light field image to obtain a low-resolution image, so as to better simulate the low-resolution light field image in real life, and then the practical application effect of the trained super-resolution reconstruction model can be improved, thereby improving the practicability of the super-resolution reconstruction of the light field image.

[0077] In the specific implementation process, the low-resolution light field image is used as the input of the spatial feature extraction module, and the corresponding 2D spatial features are extracted through the spatial feature extraction module.

[0078] The texture information of the light field is contained in different sub-aperture images, which plays a key role in reconstructing the texture details of the high-resolution image. In order to fully extract the spatial information of the light field image, the present invention designs a spatial feature extraction network structure of the light field image based on the ResRFB (ResidualRFB, ResRFB) and ResBlock (Residual Block, ResBlock) modules.

[0079] The spatial feature extraction module includes a convolutional layer, a first ResBlock module, a first ResRFB module, a second ResBlock module, and a second ResRFB module that are connected end to end in sequence (that is, the output end of the previous module is connected to the input end of the next module);

[0080] The convolutional layer is used to extract the shallow features of the light field image;

[0081] The first ResBlock module and the second ResBlock module are used to activate the extracted features; ResBlock is a feature extraction module used to obtain deeper feature representations.

[0082] The first ResRFB module and the second ResRFB module are used to expand the receptive field through dilated convolution, strengthen the correlation between channel pixels, and then extract spatial features; the output of the second ResRFB module is the final 2D spatial feature.

[0083] The overall process of the spatial feature extraction module is as follows: the low-resolution light field sub-aperture image first passes through a convolutional layer to extract shallow features, and then is input into the ResBlock and ResRFB modules that appear alternately to extract the target spatial features. The 2D spatial feature F S is composed of F c and F i :

[0084] F c ∈L H×W×C ;

[0085] F i ∈L H×W×C(i = 1, 2, … C 2 - 1);

[0086] Wherein: F c represents the central view feature of the light field image; F i represents the side view feature of the light field image; L represents the light field; H×W represents the spatial resolution of the light field; C represents the number of channels. It should be noted that F c and F i are the arranged original sub-aperture images, and F c and F i together form the sub-aperture image array.

[0087] As Figure 3 (b) shows, both the first ResBlock module and the second ResBlock module include a 3×3 convolutional layer, a Relu activation function layer, and a 3×3 convolutional layer connected end to end in sequence. The ResBlock module consists of 2 3×3 convolutional layers and one Relu layer, mainly used to activate the features extracted by ResRFB.

[0088] As Figure 3 (a) shows, both the first ResRFB module and the second ResRFB module include a first 1×1 convolutional layer (to reduce the number of channels of the feature map), three branches with their input ends respectively connected to the output end of the first 1×1 convolutional layer, a connection layer with its input ends respectively connected to the output ends of the three branches and used to connect the output features of the three branches, and a second 1×1 convolutional layer with its input end connected to the output end of the connection layer (the second 1×1 convolutional layer is used to fuse the extracted features); a Relu activation function layer is arranged between the connection layer and the second 1×1 convolutional layer. The three branches include: a first branch composed of a third 1×1 convolutional layer and a dilated convolutional layer with a dilation rate of 1, a second branch composed of a 3×3 convolutional layer and a dilated convolutional layer with a dilation rate of 3, and a third branch composed of a 5×5 convolutional layer and a dilated convolutional layer with a dilation rate of 5.

[0089] In the ResRFB module, in order to reduce the number of channels of the feature map, the low-resolution light field image first passes through a 1×1 convolutional layer; then enters the dilated convolution with dilation rates r = 1, 3, 5 for feature extraction, where r represents the parameter of the dilated convolutional layer; finally, the features extracted at each level are connected and then fused by a 1×1 convolutional kernel for the extracted features and sent to the next level. In the present invention, a residual structure is added on the basis of RFB to form the ResRFB module, which is a multi-branch structure composed of convolutional layers with different-sized convolutional kernels and dilated pooling layers, and uses dilated convolution to expand the receptive field, strengthening the correlation between channel pixels, and finally realizing the effective extraction of spatial texture information.

[0090] In order to fully extract the spatial texture features of the sub-aperture images of the light field, the present invention designs a spatial feature extraction module based on the ResRFB module and the ResBlock module according to the inherent structure of the 4D light field image, enabling the rapid and efficient extraction of the texture detail features in the light field image while ensuring the image resolution. The present invention adds a residual structure to the RFB to form the ResRFB module. The ResRFB module is a multi-branch structure composed of convolutional layers with different-sized convolutional kernels and atrous pooling layers, and uses atrous convolution to expand the receptive field, strengthening the correlation between channel pixels, fully learning the 2D spatial features of each view in the light field image, and ultimately enabling the effective extraction of spatial texture information.

[0091] In the specific implementation process, the 2D spatial features of each sub-aperture image are used as a stack input, and the 2D angular features of each sub-aperture image are extracted via the angular feature extraction module (FusionA), and the 2D angular features and 2D spatial features of each sub-aperture image are connected to generate global features.

[0092] Due to factors such as edge occlusion, the pixel distribution in each view image of the light field is uneven, but the texture detail features of a distorted view can be restored and enhanced with the help of the supplementary information of other angular views. Therefore, after the present invention extracts the spatial features of the light field image, the angular information can be further extracted, and according to the correlation between the spatial features and the angular features, the two are fused to achieve the ultimate goal of improving super-resolution. The FusionA module is for the extraction of 2D angular features, and the FusionB module is for the final fusion and enhancement of 2D angular features and 2D spatial features.

[0093] As Figure 4 shown, the angular feature extraction module includes several parallel branches for extracting new view-point features between the 2D spatial features of different sub-aperture images, a 3×3 convolutional layer with its input end connected to the output ends of several parallel branches respectively and for merging the output features of each parallel branch, and a feature connection layer for connecting the 2D angular features and 2D spatial features and generating global features.

[0094] The parallel branches include a 9×9 convolutional layer, a Relu activation function layer, a 5×5 convolutional layer, a Relu activation function layer, a 3×3 convolutional layer, and a Relu activation function layer connected end to end in sequence. Branch takes different spatial feature pair stacks as input, each branch consists of three convolutional layers, and performs 9×9, 5×5, and 3×3 depth convolutions in sequence. The feature maps generated by the Branch(i) sub-module are merged through a 3×3 convolutional layer, and finally a new image view-point containing angular features is generated. Where i takes the value of 4.

[0095] The global features are represented by the following formula:

[0096]

[0097] In the formula: F G represents the global feature; F S represents the 2D spatial feature; F A represents the 2D angular feature; Convert G (·) represents 2D convolution; concat(·) represents the concatenation (Concatenate) operation; R represents reshape.

[0098] The angular feature extraction module of the present invention searches for angular constraints between different perspectives of the light field sub-aperture image through multiple parallel branches, and then fully fuses and enhances it with the extracted 2D spatial texture features, so as to improve the super-resolution accuracy, and thus can better achieve the effective complementarity of spatial features and angular features in the light field image, which can further improve the effectiveness and accuracy of the super-resolution reconstruction of the light field image.

[0099] In the specific implementation process, the global features of each sub-aperture image are used as the input of the feature fusion and enhancement module, and 4D light field features are generated after the global features are fused and enhanced by the feature fusion and enhancement module.

[0100] In order to further utilize the 2D spatial information and 2D angular information of the light field image, F G is input into the second part of the feature fusion module, the feature fusion and enhancement module (FusionB).

[0101] As Figure 5 shown, the feature fusion and enhancement module includes a number of spatial angular convolution layers (Spatial Angular Convolution, SASConvert) connected end to end in sequence; the spatial angular convolution layer includes a spatial convolution layer, a Relu activation function layer, an angular convolution layer and a Relu activation function layer connected end to end in sequence. The FusionB module is composed of n SASConv, which alternately processes the feature information of the spatial dimension and the angular dimension with 2D convolution, and can accumulate more texture detail information on the original features without increasing the number of parameters. Compared with three-dimensional and four-dimensional convolutions, 2D convolution can significantly reduce the computational complexity.

[0102] In this embodiment, the spatial angular convolution layer is an existing mature network structure, which can be referred to the spatial angular convolution layer disclosed in "Yeung HenryWing Fung, Hou Junhui, Chen Xiaoming, et al. Light field spatial super-resolutionusing deep efficient spatial-angular Separable Convolution".

[0103] The feature fusion and enhancement module of the present invention fuses and enhances the global features through the spatial angular convolution layer, which can alternately process the feature information of the spatial dimension and the angular dimension with two-dimensional convolution. It can accumulate more texture detail information on the original features without increasing the number of parameters, and can generate 4D light field structure features that can be used for super-resolution reconstruction of light field images, thereby improving the effectiveness of super-resolution reconstruction of light field images. Compared with three-dimensional and four-dimensional convolutions, two-dimensional convolution can significantly reduce the computational complexity. The present invention makes full use of the correlation between the spatial texture information and the geometric angle information in the light field image, introduces the fusion and enhancement of the geometric angle structure of each view in the feature fusion module, and the designed network can effectively learn the redundant information in the light field image, improving the geometric consistency of the super-resolution result.

[0104] In the specific implementation process, the 4D light field feature is used as the input of the reconstruction module, and the 4D light field feature is super-resolution reconstructed by the upsampling method based on sub-pixel convolution in the reconstruction module.

[0105] By making full use of the spatial feature information and the angular feature information between the sub-aperture images of the light field, the accuracy of the reconstructed image can be further improved. The task of the reconstruction module of the present invention is to reconstruct a high-resolution sub-aperture image array according to the fused and enhanced 4D light field feature F. By using the upsampling method based on sub-pixel convolution, the reconstructed feature can be upgraded to the target resolution aH×aW, where a refers to the upsampling ratio.

[0106] In this embodiment, the upsampling method based on sub-pixel convolution adopted by the reconstruction module is an existing mature method, which can be referred to the corresponding content disclosed in "Shi Wenzhe, Caballero Jose, Huszár Ferenc, et al., Real-time singleimage and video super-resolution using an efficient sub-pixel convolutionalneural network".

[0107] The reconstruction module first performs a 1×1 convolution on the 4D light field feature map with C channels to expand its channel number to a2×C; then performs pixel recombination to generate a high-resolution feature map with a resolution a times higher; finally, performs feature fusion on the high-resolution feature map through a convolutional layer to generate a sub-aperture image array of the super-resolved light field, that is, the super-resolution reconstruction image.

[0108] In the specific implementation process, the loss function loss of the present invention is defined using the L1 norm, and the finally generated super-resolution image L SR and the original image L HR are compared, which has good robustness to outliers during the neural network training process.

[0109] The training loss is calculated by the following formula:

[0110] loss = ||L SR (u, v, s, t) - L HR (u, v, s, t)|| 1 ;

[0111] In the formula: L SR (u, v, s, t) represents the generated super-resolution reconstruction image; L HR (u, v, s, t) represents the high-resolution light field image used as a sample; (u, v) and (s, t) respectively represent the angular plane coordinates and spatial plane coordinates of the light field image.

[0112] In the present invention, the loss function of the model is defined using the L1 norm. The greatest advantage of the L1 loss function is its strong robustness. And when facing observations with large errors, the L1 loss function is not easily affected by it. At the same time, the L1 loss is more conducive to reconstructing sharp edges and has good robustness to outliers during the neural network training process.

[0113] To better illustrate the advantages of the technical solution of the present invention, the following experiment is disclosed in this embodiment.

[0114] This experiment uses 4 publicly available light field datasets to conduct a series of tests on the effectiveness of the proposed method. The datasets include: a total of 36 four-dimensional (4D) synthetic light field images from HCInew and HCIold, and a total of 390 real light field images from Stanford and EPFL captured by the Lytro Illum light field camera. Among them, 30 4D synthetic light field images and 330 real light field image data are used for training, and the remaining 66 image data are used for testing. The specific information is shown in Table 1.

[0115] Table 1 Training dataset and test dataset

[0116]

[0117] In the experiment, Bicubic was used as a benchmark and compared with the classic single image super-resolution method EDSR based on deep learning and the light field image super-resolution reconstruction methods LFBM5D, GBSR and ResLF. Among them, LFBM5D and GBSR are light field image super-resolution methods based on traditional features, and ResLF is a light field image super-resolution method based on deep learning. The corresponding experimental data are obtained based on the code and model training provided by the authors of each existing paper. The ×2 super-resolution results on the test dataset are shown in Table 2. The data in the table are the average PSNR and SSIM of all test light field images in the corresponding dataset. The bold data are the optimal data, and the underlined data are the suboptimal data.

[0118] Experimental results show that the method of the present invention performs best on four data sets, with PSNR improved by 1.42dB, 2.23dB, 2.99dB and 1.49dB respectively, and SSIM improved by 0.07%, 0.02%, 0.11% and 0.10% respectively compared with the suboptimal algorithm.

[0119] Table 2 Comparison of average PSNR (dB) / SSIM of ×2 super-resolution results of different algorithms

[0120]

[0121] also, Figure 6 , Figure 7 and Figure 8 The super-resolution reconstruction results of the scene origami, bicycles and occlusions by the method of the present invention (i.e. Propoesd in the figure) and four existing algorithms are respectively shown.

[0122] Experimental results show that the light field image reconstructed by the Bicubic method is generally blurry. The reason is that the image usually contains low-frequency components and high-frequency components. This method mainly uses the low-frequency components of the image and ignores the high-frequency components such as the pixels with dramatic changes in the image, such as the edge contour and other details of the image. The EDSR method mainly uses a single view for reconstruction, so the reconstruction effect of a single image is better, but it is not suitable for light field images because it ignores the internal connection between the light field sub-aperture images and cannot reproduce complex texture information. The GBSR method is limited by the calculation accuracy of the spatial scene prior and does not handle occlusion, blur and noise problems well. The ResLF method based on the residual network can reconstruct the light field image more realistically, but ResLF only uses partial views to super-resolve different images, and does not make full use of the spatial and angular information of the light field sub-aperture image array. Therefore, the reconstruction quality of its non-central view is relatively low, and the occluded part is seriously distorted.

[0123] The super-resolution reconstruction result of the method of the present invention for light field images is closer to the Ground Truth, with clearer image textures and edges, better restoration of detail features, and the ability to reconstruct all sub-aperture image arrays simultaneously. The super-resolution quality on each viewpoint image is overall better than other methods. The reason is that the method of the present invention utilizes the feature information of all sub-aperture images in the light field image, fully extracts and fuses the spatial information and angular information between different views, and can reconstruct the texture details in the image more realistically. At the same time, the extracted global features can better improve problems such as blurred distortion caused by edge occlusion.

[0124] To further verify the computational efficiency of the method of the present invention, Table 3 shows the number of Params and the amount of FLOPs used by various methods. It can be seen that on the basis of ensuring the quality of the light field super-resolution reconstruction image, the method of the present invention consumes approximately 7 times less Params than EDSR and is also significantly lower than the method of ResLF. The amount of FLOPs is approximately 20 times less than EDSR and is comparable to ResLF.

[0125] Table 3 Paramas and FLOPs of Different Algorithms

[0126]

[0127] In summary, through experimental testing and analysis of 2 groups of 4D synthetic light field images and 2 groups of real light field image datasets captured by LytroIllum light field cameras in this experiment, it can be known that the method of the present invention shows great advantages in both subjective vision and objective performance. The super-resolution reconstruction quality of the light field image is high, the restoration effect of edge contour details is good, and high PSNR and SSIM performance are obtained with a low computational cost, which is better than 5 comparison algorithms.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Those of ordinary skill in the art should understand that any modifications or equivalent replacements to the technical solutions of the present invention without departing from the purpose and scope of the technical solutions should be covered by the scope of the claims of the present invention.

Claims

1. A super-resolution reconstruction method for light field images that fuses spatial and angular features, characterized in that, it includes: S1: Obtain the target light field image to be reconstructed; S2: Input the target light field image into the trained super-resolution reconstruction model to output the corresponding super-resolution reconstructed image; The super-resolution reconstruction model is trained through the following steps: S201: Convert the high-resolution light field image used as a sample into a low-resolution light field image and use it as the model input; S202: Extract the 2D spatial features of each sub-aperture image in the low-resolution light field image; In step S202, the low-resolution light field image is used as the input of the spatial feature extraction module, and the corresponding 2D spatial features are extracted through the spatial feature extraction module; among them, the spatial feature extraction module includes a convolutional layer, a first ResBlock module, a first ResRFB module, a second ResBlock module, and a second ResRFB module connected end to end in sequence; The convolutional layer is used to extract the shallow features of the low-resolution light field image; The first ResBlock module and the second ResBlock module are used to activate the extracted features; The first ResRFB module and the second ResRFB module are used to expand the receptive field through dilated convolution, strengthen the correlation between channel pixels, and then extract the corresponding spatial features; the output of the second ResRFB module is the final 2D spatial feature; Both the first ResRFB module and the second ResRFB module include a first 1×1 convolutional layer, three branches with their input ends respectively connected to the output end of the first 1×1 convolutional layer, a connection layer with its input ends respectively connected to the output ends of the three branches and used to connect the output features of the three branches, and a second 1×1 convolutional layer with its input end connected to the output end of the connection layer; a Relu activation function layer is arranged between the connection layer and the second 1×1 convolutional layer; The three branches include: a first branch composed of a third 1×1 convolutional layer and a dilated convolutional layer with a dilation rate of 1, a second branch composed of a 3×3 convolutional layer and a dilated convolutional layer with a dilation rate of 3, and a third branch composed of a 5×5 convolutional layer and a dilated convolutional layer with a dilation rate of 5; On the basis of RFB, a residual structure is added to form a ResRFB module, which is a multi-branch structure composed of convolutional layers with different-sized convolutional kernels and dilated pooling layers, and dilated convolution is used to expand the receptive field, strengthen the correlation between channel pixels, learn the 2D spatial features of each view in the light field image, and effectively extract the spatial texture information; S203: Extract the 2D angular features of each sub-aperture image on the basis of the 2D spatial features, and then connect the 2D spatial features and 2D angular features of each sub-aperture image to generate the corresponding global features; In step S203, the 2D spatial features of each sub-aperture image are used as the stack input, the 2D angular features of each sub-aperture image are extracted through the angular feature extraction module, and the 2D angular features and 2D spatial features of each sub-aperture image are connected to generate global features; The angular feature extraction module includes several parallel branches for extracting new view point features between 2D spatial feature pairs of different sub-aperture images, a 3×3 convolutional layer with its input end connected to the output ends of several parallel branches and used for merging the output features of each parallel branch, and a feature connection layer for connecting 2D angular features and 2D spatial features and generating global features; Search for angular constraints between different perspectives of the light field sub-aperture images through multiple parallel branches; the parallel branches include a 9×9 convolutional layer, a Relu activation function layer, a 5×5 convolutional layer, a Relu activation function layer, a 3×3 convolutional layer, and a Relu activation function layer connected end to end in sequence; S204: Fuse and enhance the global features of each sub-aperture image to generate 4D light field features of the light field image; S205: Perform super-resolution reconstruction based on the 4D light field features of the light field image to generate a super-resolution reconstructed image; S206: Calculate the training loss based on the super-resolution reconstructed image and the high-resolution light field image used as a sample, and optimize the parameters of the super-resolution reconstruction model based on the training loss; S207: Repeat steps S201 to S206 until the super-resolution reconstruction model converges; S3: Use the output super-resolution reconstructed image as the super-resolution reconstruction result of the target light field image.

2. The light field image super-resolution reconstruction method for fusing spatial and angular features according to claim 1, characterized in that: In step S201, perform bicubic downsampling on the high-resolution light field image to obtain a low-resolution light field image.

3. The light field image super-resolution reconstruction method for fusing spatial and angular features according to claim 1, characterized in that: Both the first ResBlock module and the second ResBlock module include a 3×3 convolutional layer, a Relu activation function layer, and a 3×3 convolutional layer connected end to end in sequence.

4. The light field image super-resolution reconstruction method for fusing spatial and angular features according to claim 1, characterized in that: The global feature of the sub-aperture image is represented by the following formula: Where: F G represents the global feature; F S represents the 2D spatial feature; F A represents a 2D angular feature; Convert G (·) represents a 2D convolution operation; concat(·) represents a feature concatenation operation; R represents a dimensionality transformation.

5. The light field image super-resolution reconstruction method for fusing spatial and angular features according to claim 1, characterized in that: In step S204, use the global features of each sub-aperture image as the input of the feature fusion and enhancement module, and generate 4D light field features after fusing and enhancing the global features by the feature fusion and enhancement module; The feature fusion and enhancement module includes several spatial angular convolutional layers connected end to end in sequence, and the spatial angular convolutional layer includes a spatial convolutional layer, a Relu activation function layer, an angular convolutional layer, and a Relu activation function layer connected end to end in sequence.

6. The light field image super-resolution reconstruction method for fusing spatial and angular features according to claim 1, characterized in that: In step S205, use the 4D light field features as the input of the reconstruction module, and perform super-resolution reconstruction on the 4D light field features by the reconstruction module based on the sub-pixel convolution upsampling method; The reconstruction module first performs a 1×1 convolution on the 4D light field feature map with C channels to expand its channel number to 2×C; then performs pixel recombination to generate a high-resolution feature map with a resolution a times higher; finally, performs feature fusion on the high-resolution feature map through a convolutional layer to generate a light field sub-aperture image array after super-resolution, that is, the super-resolution reconstruction image.

7. The light field image super-resolution reconstruction method for fusing spatial and angular features according to claim 1, characterized in that: in step S206, the training loss is calculated by the following formula: loss = ||L SR (u, v, s, t)-L HR (u, v, s, t)|| 1 ; Where: L SR (u, v, s, t) represents the generated super-resolution reconstructed image; L HR (u, v, s, t) represents the high-resolution light field image used as a sample; (u, v) and (s, t) respectively represent the angular plane coordinates and the spatial plane coordinates of the light field image.

Images