Spatial super-resolution method of light field images based on space-angle decoupling mechanism

Abstract

The present invention discloses a light field image spatial super-resolution method based on a space-angle decoupling mechanism, which uses a trained spatial super-resolution reconstruction model to generate a high-resolution light field image from an original light field image data set; the spatial super-resolution reconstruction model maps the input light field image into shallow features through a shallow feature extraction network, extracts multi-scale space-angle feature maps through a spatial feature extractor and an angle feature extractor, extracts the epipolar plane image feature map through an epipolar plane image feature extractor, and then performs upsampling based on the fusion results of the two, and performs addition operation on the upsampled output and the feature map obtained by the bicubic interpolation method of the input light field image to fuse and generate a spatial super-resolution reconstruction image of the light field image. The present invention fully obtains the deep intrinsic features of the light field based on the asymmetry and non-local correlation of the space-angle, thereby obtaining a spatial super-resolution reconstruction image of the light field image.

Description

Technical Field

[0001] The invention relates to the technical field of light field image super-resolution reconstruction, and in particular to a light field image spatial super-resolution method based on a space-angle decoupling mechanism. Background Art

[0002] Light field (LF) imaging records both the intensity and direction of light. Compared with traditional imaging methods, LF imaging has potential application prospects in vision fields such as depth estimation, virtual reality, and three-dimensional (3D) reconstruction. However, due to the limitations of LF imaging sensors, the spatial resolution of LF images is low, which hinders the practical application of LF imaging. Therefore, it is of great practical significance to develop robust and efficient LF image spatial super-resolution (SR) methods.

[0003] Traditional LF image spatial SR methods mainly use manually extracted image priors to improve resolution, but the reconstructed image quality is poor. In recent years, with the rapid development of deep neural networks, many learning-based methods have been applied to the field of LF image spatial SR. Four-dimensional (4D) convolution is the simplest method to process 4D LF data, but its computational complexity and memory requirements are high. Therefore, pseudo-4D models that use two-dimensional (2D), 3D convolution or Transformer to simulate 4D convolution have become the mainstream method. In these methods, the space-angle decoupling mechanism usually converts the four-dimensional 4D light field image into three 2D subspace images for processing, namely, sub-aperture image (SAI), macro-pixel image (MacPI) and epipolar-plane image (EPI); on this basis, various network modules are designed to extract the structural prior information of the 4D light field image from the three 2D subspace images. On the one hand, the existing methods treat the spatial dimension and angular dimension in the LF image equally, resulting in the same receptive field range in the two dimensions. However, the spatial and angular dimensions of LF images are asymmetric. For example, the spatial resolution of LF images taken by the Lytro IIIum camera is 37×540, while the angular resolution is only 14×14. Figure 3The difference in information volume between a single SAI and a single MacPI in the figure also intuitively shows the spatial-angular asymmetry of the LF image. Therefore, using the same receptive field to process the spatial and angular dimension information of the LF image usually leads to insufficient extraction of spatial information and redundancy of angular information. On the other hand, the spatial-angular disentanglement mechanism proposed by existing methods usually uses a single convolution cascade to process EPI, which fails to fully capture the long-range spatial-angular feature information of the LF image due to the limitations of the convolution operation. However, since the slope of each epipolar line in the EPI shows a strong correlation with the disparity of the scene, the change of the epipolar line reflects the non-local correlation of the spatial-angular space of the LF image. Therefore, by expanding the receptive field range of the EPI, effective modeling of the non-local correlation of the spatial-angular space can be achieved.

[0004] Therefore, how to effectively utilize the spatial-angular non-stacked and non-local correlation of LF images to design an efficient SR method to fully extract the feature information of the three 2D subspace images has become a problem that technicians in this field need to solve urgently. Summary of the invention

[0005] In view of the deficiencies of the above-mentioned prior art, the present invention provides a light field image spatial super-resolution method based on a space-angle decoupling mechanism. According to the space-angle asymmetry and non-local correlation, by constructing spatial, angular and extreme plane image feature extractors, different receptive fields are used to fully acquire feature information in sub-aperture images, macro-pixel images and epipolar plane images, and deep-level intrinsic features of the light field are obtained, thereby obtaining a spatial super-resolution reconstructed image of the light field image.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The light field image spatial super-resolution method based on the space-angle decoupling mechanism includes:

[0008] S1: Acquire the target light field image to be reconstructed;

[0009] S2: input the target light field image into the trained spatial super-resolution reconstruction model, and output the spatial super-resolution reconstructed image of the target light field image;

[0010] The training steps of the spatial super-resolution reconstruction model are as follows:

[0011] S201: Converting a high-resolution light field image into a low-resolution light field image as a training sample, and inputting the result into a spatial super-resolution reconstruction model; the spatial super-resolution reconstruction model includes a shallow feature extraction network, a multi-dimensional feature extraction network, and an upsampling reconstruction network;

[0012] S202: The shallow feature extraction network maps the light field image input by the model into shallow features as an input feature map of the multidimensional feature extraction network;

[0013] S203: The multidimensional feature extraction network includes a spatial feature extractor, an angular feature extractor and an epipolar plane image feature extractor, which respectively extract spatial features, angular features and epipolar plane image features from the subspace image, macropixel image and epipolar plane image of the input feature map, and fuse them to generate 4D light field features of the light field image as input of the upsampling reconstruction network;

[0014] S204: the upsampling reconstruction network performs upsampling and reconstruction processing on the 4D light field features of the light field image to generate a spatial super-resolution reconstructed image of the light field image;

[0015] S205: calculating a training loss according to the spatial super-resolution reconstructed image of the light field image as a training sample and its corresponding high-resolution light field image, and optimizing parameters of the spatial super-resolution reconstruction model based on the training loss;

[0016] S206: Repeat steps S201 to S205 until the spatial super-resolution reconstruction model converges or reaches a preset number of iterations;

[0017] S3: The spatial super-resolution reconstructed image output by the spatial super-resolution reconstruction model is used as the super-resolution reconstruction result of the target light field image.

[0018] As a preferred solution, in step S201, the high-resolution light field image is cropped, and the cropped light field image is downsampled using a bicubic interpolation method to obtain a low-resolution light field image as a training sample.

[0019] As a preferred solution, in step S203, the multidimensional feature extraction network includes a plurality of multidimensional feature extraction modules cascaded in sequence, each of which includes a top branch, a bottom branch and a connection layer;

[0020] The top branch includes a cascaded spatial feature extractor and an angular feature extractor. The spatial feature extractor is used to extract the spatial features of the sub-aperture image in the input feature map. The obtained spatial feature map is input to the angular feature extractor to further extract the angular features of its macro-pixel image, thereby obtaining a multi-scale spatial-angular feature map of the input feature map.

[0021] The bottom branch includes a horizontal epipolar plane image feature extractor and a vertical epipolar plane image feature extractor which are cascaded in sequence by weight sharing, the horizontal epipolar plane image feature extractor is used to extract horizontal epipolar plane image features of the input feature map, the obtained horizontal epipolar plane image feature map is input to the vertical epipolar plane image feature extractor for further extracting vertical epipolar plane image features, and obtaining an epipolar plane image feature map of the input feature map;

[0022] The connection layer is used to splice the obtained multi-scale spatial-angular feature map and the epipolar plane image feature map, and after activation processing through the convolution activation unit, it is fused with the input feature map to generate the 4D light field features of the light field image as the input of the upsampling reconstruction network.

[0023] As a preferred solution, the spatial feature extractor includes a dense convolutional network, a convolutional activation unit and a Swin Transformer module connected in sequence;

[0024] The dense convolution network includes Ns cascaded spatial convolution units, and the spatial convolution unit includes a cascaded 3×3 convolution layer and a leaky ReLU activation function layer; wherein the input of the spatial feature extractor is respectively used as the input of the Ns spatial convolution units, and the output of the previous spatial convolution unit is respectively used as the input of the subsequent spatial convolution unit, and the output of the Ns spatial convolution units is concatenated as the output of the dense convolution network, and the output of the dense convolution network is used as the input of the convolution activation unit;

[0025] The convolution activation unit includes a cascaded 1×1 convolution layer and a Leaky ReLU activation function layer;

[0026] The Swin Transformer module includes a cascaded layer-normalized multi-head attention unit and a layer-normalized perception unit, wherein the layer-normalized multi-head attention unit includes a cascaded layer-normalized layer and a moving window multi-head self-attention module, and the layer-normalized perception unit includes a cascaded layer-normalized layer and a multi-layer perceptron; wherein the input of the Swin Transformer module is used as the input of the layer-normalized multi-head attention unit, the output of the layer-normalized multi-head attention unit is added to the input of the Swin Transformer module as the input of the layer-normalized perception unit, and the output of the layer-normalized perception unit is added to the input of the layer-normalized perception unit as the output of the Swin Transformer module;

[0027] In the spatial feature extractor, the shallow feature shaping of the light field image is first converted into a stacked array of sub-aperture images as the input of the dense convolutional network. The output of the dense convolutional network is used as the input of the convolutional activation unit to generate a multi-scale spatial feature map as the output of the convolutional activation unit. The multi-scale spatial feature map is used as the input of the Swin Transformer module, and the output of the Swin Transformer module is used as the overall output of the spatial feature extractor.

[0028] As a preferred solution, the processing process of the spatial feature extractor is expressed by the following formula:

[0029]

[0030] F LN =LN(F D );

[0031] F S(W) =SW-MSA(F LN )+F D ;

[0032]

[0033] In the formula, S i represents the i-th densely connected 3×3 spatial convolutional layer, Conv 1×1 represents a 1×1 convolutional layer, cat represents channel cascade, [·] represents cascade, and F D Represents dense convolutional networks and Conv 1×1 The generated multi-scale spatial features, represents the characteristics of the subaperture image stack array, F LN Represents the output of the normalized layer in the multi-head attention unit, F S(W) The input of the normalized perception unit of the representation layer, Represents the output of the spatial feature extractor; SW-MSA represents the moving window multi-head self-attention module, MLP represents the multi-layer perceptron, and LN represents the regularization processing of the layer normalization layer.

[0034] As a preferred solution, the angle feature extractor includes Na cascaded angle convolution units, and the angle convolution unit includes a 3×3 angle convolution layer and a Leaky ReLU activation function layer; in the angle feature extractor, the output feature of the spatial feature extractor is shaped and converted into a macro-pixel image stack array as the input of the first angle convolution unit, and the output of the Nath angle convolution unit is used as the overall output of the angle feature extractor; the processing process of the angle feature extractor is expressed by the following formula:

[0035]

[0036] In the formula, represents the output of the angle feature extractor, A j represents the jth 3×3 angle convolution layer, Characterizes a macropixel image stack array.

[0037] As a preferred solution, the horizontal epipolar plane image feature extractor and the vertical epipolar plane image feature extractor in the bottom branch are both epipolar plane image feature extractors, and the epipolar plane image feature extractors include three dilated convolution units and a convolution activation unit connected in parallel, wherein the parallel connection result of the dilated convolution units is cascaded with the convolution activation unit;

[0038] The three dilated convolution units connected in parallel are respectively a first dilated convolution unit, a second dilated convolution unit and a third dilated convolution unit; the first dilated convolution unit includes a 3×3 convolution layer with a dilation rate of 1 and a Leaky ReLU activation function layer, the second dilated convolution unit includes a 3×3 convolution layer with a dilation rate of 2, and the third dilated convolution unit includes a 3×3 convolution layer with a dilation rate of 4; the outputs of the three dilated convolution units connected in parallel are spliced ​​and used as the input of the convolution activation unit, and the output of the convolution activation unit is then added to the shallow features of the input light field image as the output of the epipolar plane image feature extractor;

[0039] In the bottom branch, the shallow features of the light field image are first shaped and converted into a stacked array of horizontal epipolar plane images as the input of the horizontal epipolar plane image feature extractor, and the horizontal epipolar plane image features are extracted. Then, the output horizontal epipolar plane image features are shaped and converted into a stacked array of vertical epipolar plane images as the input of the vertical epipolar plane image feature extractor, and the epipolar plane image feature map is extracted as the output of the bottom branch in the multidimensional feature extraction network.

[0040] As a preferred solution, the processing of the horizontal epipolar plane image feature extractor and the vertical epipolar plane image feature extractor are respectively expressed as follows:

[0041]

[0042] In the formula, and denote the outputs of the horizontal epipolar plane image feature extractor and the vertical epipolar plane image feature extractor respectively; Characterizes the stacked array of horizontal epipolar plane images, represents the features of the stacked array of vertical epipolar plane images, and E represents the epipolar plane image feature extractor.

[0043] As a preferred solution, in step S204, the upsampling reconstruction network upsamples the 4D light field features of the light field image, and then adds the upsampling to the feature map obtained by bicubic interpolation of the light field image input by the model to generate a spatial super-resolution reconstructed image of the light field image output by the model;

[0044] The upsampling reconstruction network includes a first 1×1 convolutional layer, a pixelshuffle layer, and a second 1×1 convolutional layer cascaded in sequence; wherein the input of the upsampling reconstruction network is used as the input of the first 1×1 convolutional layer, and the number of input feature channels is increased to α 2 C. Then, the output of the first 1×1 convolutional layer is used as the input of the pixelshuffle layer to reshape the output features of the first 1×1 convolutional layer from the channel dimension to the spatial dimension to generate super-resolution features αH×αW. Finally, the super-resolution features αH×αW are used as the input of the second 1×1 convolutional layer for super-resolution feature refinement, and the output is the output of the upsampling reconstruction network; α is the spatial resolution magnification factor.

[0045] As a preferred solution, in step S205, the training loss is calculated by the following formula:

[0046]

[0047] Where, L 1 represents the loss function, Represents the low-resolution light field image of the mth training sample The spatial super-resolution reconstructed image, Represents the low-resolution light field image of the mth training sample The corresponding high-resolution light field image, M represents the number of training samples contained in the training sample set.

[0048] Compared with the prior art, the present invention has the following technical effects:

[0049] 1. In the method of the present invention, a spatial feature extractor and an angular feature extractor are constructed respectively, and different receptive fields are used to asymmetrically extract spatial and angular features, thereby maximizing the utilization of feature information in sub-aperture images and macro-pixel images, thereby solving the problem that the existing light field image spatial super-resolution method ignores the spatial-angular asymmetry, resulting in insufficient extraction of feature information in sub-aperture images and macro-pixel images.

[0050] 2. In the method of the present invention, an epipolar plane image feature extractor is constructed. By using different filling rates and dilation rates, the epipolar plane feature extractor has receptive fields of different scales, so that it can effectively capture long-distance space-angle information and achieve effective modeling of space-angle non-local correlation. This solves the problem that the existing light field image spatial super-resolution method uses conventional convolution to extract feature information of the epipolar plane image, ignores the space-angle non-local correlation, and fails to fully utilize the long-distance space-angle feature information.

[0051] 3. The super-resolution reconstruction results of light field images obtained by the method proposed in the present invention are superior to similar methods in the prior art in terms of visual quality and quantitative quality indicators, providing a more effective technical auxiliary means for the wider practical application of light field images. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0053] Figure 1 It is a flow chart of the light field image spatial super-resolution method based on the space-angle decoupling mechanism of the present invention;

[0054] Figure 2 Schematic diagram of the overall architecture of the spatial super-resolution reconstruction model in the method of the present invention;

[0055] Figure 3 The following are examples of the transformation of three 2D subspace images of a light field image: subaperture image, macropixel image, and epipolar plane image.

[0056] Figure 4 This is an example of the 2×SR reconstruction results of different methods in the experimental example;

[0057] Figure 5 This is an example of the 4×SR reconstruction results of different methods in the experimental example;

[0058] Figure 6 This is the PSNR distribution diagram of different methods in the 2×SR reconstruction task under 5×5 viewpoints in the experimental example;

[0059] Figure 7 This is the depth estimation diagram of the 4×SR reconstruction results of the scene HCIold_buddha using different methods in the experimental example;

[0060] Figure 8 Comparison diagram of the refocused images generated by the method of the present invention at three different focal planes (top) and the original high-resolution HR light field image (bottom) in the experimental example. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0062] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] Light Field (LF) imaging records both the intensity and angle information of light and has potential applications in many fields. However, some commercial microlens cameras sacrifice spatial resolution to obtain higher angular resolution, resulting in low spatial resolution of the LF images, which limits the practical application of LF imaging.

[0064] In order to overcome this limitation, the present invention proposes a light field image spatial super-resolution method based on a space-angle decoupling mechanism. The method process is as follows: Figure 1 As shown, including:

[0065] S1: Acquire the target light field image to be reconstructed;

[0066] S2: input the target light field image into the trained spatial super-resolution reconstruction model, and output the spatial super-resolution reconstructed image of the target light field image;

[0067] The training steps of the spatial super-resolution reconstruction model are as follows:

[0068] S201: Converting a high-resolution light field image into a low-resolution light field image as a training sample, and inputting the result into a spatial super-resolution reconstruction model; the spatial super-resolution reconstruction model includes a shallow feature extraction network, a multi-dimensional feature extraction network, and an upsampling reconstruction network;

[0069] S202: The shallow feature extraction network maps the light field image input by the model into shallow features as an input feature map of the multidimensional feature extraction network;

[0070] S203: The multidimensional feature extraction network includes a spatial feature extractor, an angular feature extractor and an epipolar plane image feature extractor, which respectively extract spatial features, angular features and epipolar plane image features from the subspace image, macropixel image and epipolar plane image of the input feature map, and fuse them to generate 4D light field features of the light field image as input of the upsampling reconstruction network;

[0071] S204: the upsampling reconstruction network performs upsampling and reconstruction processing on the 4D light field features of the light field image to generate a spatial super-resolution reconstructed image of the light field image;

[0072] S205: calculating a training loss according to the spatial super-resolution reconstructed image of the light field image as a training sample and its corresponding high-resolution light field image, and optimizing parameters of the spatial super-resolution reconstruction model based on the training loss;

[0073] S206: Repeat steps S201 to S205 until the spatial super-resolution reconstruction model converges or reaches a preset number of iterations;

[0074] S3: The spatial super-resolution reconstructed image output by the spatial super-resolution reconstruction model is used as the super-resolution reconstruction result of the target light field image.

[0075] The image super-resolution (SR) described in the present invention refers to a technology that uses image processing and machine learning technology to reconstruct a high-resolution (HR) image from an existing low-resolution (LR) image. Specifically, a light field image can be represented as three 2D subspace image forms: a sub-aperture image (SAI), a macro-pixel image (MacPI), and an epipolar plane image (EPI), and the three 2D subspace image forms can be converted into each other. The present invention represents a light field image as a sub-aperture image array. The light field can be represented as I LR ∈R B×U×V×H×W , where C is the number of channels, B is the batch size, U and V are the angular resolutions, and H and W are the spatial resolutions. The goal of light field image super-resolution reconstruction is to reconstruct a low-resolution (LR) light field image I LR ∈R B×U×V×H×W Reconstruct the super-resolution (SR) light field image I SR ∈R B×U×V×αH×αW , α is the spatial resolution magnification.

[0076] Through the above process of the method of the present invention, it can be seen that, considering the spatial-angular asymmetry, the present invention designs a spatial feature extractor (SpatialFeature Extractor, SFE) and an angular feature extractor (Angular Feature Extractor, AFE) for processing sub-aperture images (SAI) and macro-pixel images (MacpI) respectively. The asymmetric structure of these two feature extractors ensures that spatial features and angular features with different receptive fields can be obtained; on the other hand, based on the non-local correlation of spatial angles, the present invention proposes an epipolar plane image feature extractor (Epipolar-Plane Image Feature Extractor, AFE). The invention discloses a method for deeply extracting epipolar plane image feature information from an epipolar plane image (EPI) by using an extractor (AFE) to deeply extract epipolar plane image feature information from an epipolar plane image (EPI); and in the method of the invention, three extractors (SFE, AFE and EFE) are combined and packaged to form a multidimensional feature extraction module, so as to realize efficient processing of three 2D subspace images (SAI, MacPI and EPI) of square image conversion, and a multidimensional feature extraction network is formed by cascading multiple multidimensional feature extraction modules to deeply explore the spatial, angular and epipolar plane feature information of light field images, obtain deep intrinsic features of the light field, and thus realize high-quality reconstruction of super-resolution light field images.

[0077] In general, the present invention aims at the unresolved spatial-angular asymmetry and non-local correlation problems in the existing LF image spatial SR methods, and proposes a light field image spatial super-resolution method based on the spatial-angular decoupling mechanism. On the one hand, it solves the asymmetry problem of spatial and angular dimensions by asymmetrically extracting the feature information of SAI and MacPI, thereby effectively extracting the intrinsic features of the LF image. On the other hand, it expands the EPI receptive field range to achieve effective modeling of spatial-angular non-local correlation, solving the problem that the existing light field image spatial super-resolution method uses conventional convolution to extract the feature information of the epipolar plane image, ignoring the spatial-angular non-local correlation, resulting in the failure to fully utilize the long-distance spatial-angular feature information. Experimental results on real-world and synthetic light field image datasets show that the super-resolution reconstruction results of light field images obtained by the method proposed in the present invention are superior to similar methods in the prior art in terms of visual quality and quantitative quality indicators, providing more effective technical auxiliary means for the wider practical application of light field images.

[0078] In order to better introduce the technical solution of the present invention, a more detailed description is given in the following parts.

[0079] 1. Image Preprocessing

[0080] In specific application implementations, the light field image data used for training and testing the spatial super-resolution reconstruction model can be selected from public light field image databases, such as EPFL, HCI_new, HCI_old, INRIA_Lytro, and Stanford_Gantry. After obtaining high-resolution light field image data from the public light field image database, the high-resolution light field image is cropped, and the cropped light field image is downsampled using the bicubic interpolation method to obtain a low-resolution light field image as a training sample. For example, in the example test, 144 scenes were selected from the above-mentioned light field image database for training and 23 scenes for testing. The original angular resolution of all the selected high-resolution light field images was 9×9. During training and testing, the angular resolution of the original light field images was cropped to 5×5, and the spatial resolution of the original light field images was cropped to 64×64 image blocks with a stride of 32. The spatial resolution of the image blocks was then downsampled to 32×32 using bicubic interpolation, so that the low-resolution light field images obtained by arranging and converting the high-resolution light field images used for training and testing scenes were used as training samples and test samples, respectively.

[0081] 2. Spatial Super-resolution Reconstruction Model

[0082] like Figure 2 As shown in (a), the overall architecture of the spatial super-resolution reconstruction model proposed in the method of the present invention includes a shallow feature extraction network (Shallow Feature Extraction), a multidimensional feature extraction network (Spa-Ang-EPI Feature Extraction & Fusion) and an upsampling reconstruction network (Feature Reconstruction). Among them, the shallow feature extraction network (Shallow Feature Extraction) is used to map the low-resolution light field image input by the model into shallow features as the input feature map of the multidimensional feature extraction network; the multidimensional feature extraction network (Spa-Ang-EPI Feature Extraction & Fusion) includes a spatial feature extractor, an angle feature extractor and an epipolar plane image feature extractor, which are used to extract spatial features, angle features and epipolar plane image features from the subspace image, macropixel image and epipolar plane image of the input feature map respectively, and fuse them to generate 4D light field features of the light field image as the input of the upsampling reconstruction network; the upsampling reconstruction network (Feature Reconstruction) performs upsampling and reconstruction processing on the 4D light field features of the light field image to generate a spatial super-resolution reconstructed image of the light field image.

[0083] Next, each network layer in the spatial super-resolution reconstruction model is explained in detail.

[0084] 3. Shallow Feature Extraction Network

[0085] In the spatial super-resolution reconstruction model, in order to facilitate the subsequent light field image feature extraction, the present invention designs an initialized shallow feature extraction network to convert the original low-resolution light field image I input to the spatial super-resolution reconstruction model into LR ∈R B×U×V×H×W Mapping to shallow features Among them, C is the number of channels, B is the batch size, U and V are the angular resolutions, and H and W are the spatial resolutions. The shallow feature extraction network includes a cascaded 3×3 convolution layer and a LeakyReLU activation function layer.

[0086] 4. Multidimensional Feature Extraction Network

[0087] In order to further enhance the feature extraction of three 2D subspace images, the present invention encapsulates the spatial feature extractor (Spatial Feature Extractor, SFE), the angular feature extractor (Angular Feature Extractor, AFE) and the epipolar plane image feature extractor (EPI Feature Extractor, EFE) into a multi-dimensional feature extraction network (Multi-Dimensional Feature Extraction Module, MDFEM) through parallel residual connections.

[0088] like Figure 2 As shown in (b), the multidimensional feature extraction network MDFEM includes a top branch, a bottom branch and a connection layer.

[0089] Among them, the top branch includes a cascaded spatial feature extractor SFE and an angular feature extractor AFE. The spatial feature extractor SFE is used to extract the spatial features of the sub-aperture image in the input feature map. The obtained spatial feature map is input to the angular feature extractor AFE for further extracting the angular features of its macro-pixel image, and obtaining a multi-scale spatial-angular feature map of the input feature map. The bottom branch includes a horizontal epipolar plane image feature extractor and a vertical epipolar plane image feature extractor cascaded in sequence by weight sharing. The horizontal epipolar plane image feature extractor is used to extract the horizontal epipolar plane image features of the input feature map. The obtained horizontal epipolar plane image feature map is input to the vertical epipolar plane image feature extractor for further extracting the vertical epipolar plane image features, and obtaining the epipolar plane image feature map of the input feature map. The connection layer is used to splice the obtained multi-scale spatial-angular feature map and the epipolar plane image feature map, and after activation processing by the convolution activation unit, it is fused with the input feature map to generate the 4D light field features of the light field image as the input of the upsampling reconstruction network.

[0090] It can be seen that the top branch of the multi-dimensional feature extraction network MDFEM consists of a cascade of SFE and AFE to deeply explore the spatial-angular features; at the same time, the bottom branch integrates two cascaded weight-sharing epipolar plane image feature extractors EFE, namely horizontal epipolar plane image feature extractors and vertical epipolar plane image feature extractors, to extract horizontal and vertical EPI features. Then, the connection layer concatenates the multi-scale spatial-angular feature map and epipolar plane image feature map extracted by the two branches, and fuses them through a 1×1 2D convolution layer in the convolution activation unit, and its output is added to the shallow features of the input to enhance the representation of light field features. This collaborative design enables the model to fully process three 2D subspace images.

[0091] In order to solve the problem of space-angle asymmetry in LF image space SR, the present invention constructs a spatial feature extractor and an angular feature extractor to asymmetrically extract spatial and angular features from a sub-aperture image (SAI) and a macro-pixel image (MacPI), respectively; in order to solve the problem of space-angle non-local correlation in LF image space SR, the present invention constructs an epipolar plane image feature extractor to fully extract feature information from an epipolar plane image (EPI).

[0092] The spatial feature extractor, angle feature extractor and epipolar plane image feature extractor are introduced in detail below.

[0093] 4.1 Spatial Feature Extractor

[0094] like Figure 2 As shown in (c), the spatial feature extractor SFE includes a dense convolutional network, a convolutional activation unit, and a Swin Transformer module connected in sequence;

[0095] The dense convolution network includes Ns cascaded spatial convolution units, and the spatial convolution unit includes a cascaded 3×3 convolution layer and a leaky ReLU activation function layer; wherein the input of the spatial feature extractor is respectively used as the input of the Ns spatial convolution units, and the output of the previous spatial convolution unit is respectively used as the input of the subsequent spatial convolution unit, and the output of the Ns spatial convolution units is concatenated as the output of the dense convolution network, and the output of the dense convolution network is used as the input of the convolution activation unit;

[0096] The convolution activation unit includes a cascaded 1×1 convolution layer and a Leaky ReLU activation function layer;

[0097] The Swin Transformer module includes a cascaded layer-normalized multi-head attention unit and a layer-normalized perception unit, wherein the layer-normalized multi-head attention unit includes a cascaded layer-normalized layer and a moving window multi-head self-attention module, and the layer-normalized perception unit includes a cascaded layer-normalized layer and a multi-layer perceptron; wherein the input of the Swin Transformer module is used as the input of the layer-normalized multi-head attention unit, the output of the layer-normalized multi-head attention unit is added to the input of the Swin Transformer module as the input of the layer-normalized perception unit, and the output of the layer-normalized perception unit is added to the input of the layer-normalized perception unit as the output of the Swin Transformer module;

[0098] In the spatial feature extractor, the shallow features of the light field image are first Reshape conversion to sub-aperture image stack array As the input of the dense convolutional network, the output of the dense convolutional network is used as the input of the convolutional activation unit to generate a multi-scale spatial feature map as the output of the convolutional activation unit, and the multi-scale spatial feature map is used as the input of the Swin Transformer module. The output of the Swin Transformer module is used as the overall output of the spatial feature extractor. The light field image shaping is converted into a stacked array of sub-aperture images (SAI) as shown in Figure 3 (a) shown.

[0099] The processing of the spatial feature extractor can be expressed by the following formula:

[0100]

[0101] F LN =LN(F D );

[0102] F S(W) =SW-MSA(F LN )+F D ;

[0103]

[0104] In the formula, S i represents the i-th densely connected 3×3 spatial convolutional layer, Conv 1×1 represents a 1×1 convolutional layer, cat represents channel cascade, [·] represents cascade, and F D Represents dense convolutional networks and Conv 1×1 The generated multi-scale spatial features, represents the characteristics of the subaperture image stack array, F LNRepresents the output of the normalized layer in the multi-head attention unit, F S(W) The input of the normalized perception unit of the representation layer, Represents the output of the spatial feature extractor; SW-MSA represents the moving window multi-head self-attention module, MLP represents the multi-layer perceptron, and LN represents the regularization processing of the layer normalization layer.

[0105] like Figure 2 As shown in (e), for the calculation of the shift window multi-head self-attention (SW-MSA), the shift window multi-head self-attention module first divides the input feature map of size H×W×C into multiple local windows of size M×M×C, and then performs self-attention calculation on each local window; the specific self-attention calculation formula is expressed as:

[0106]

[0107] In the formula, F att (Q s ,K s ,V s ) represents the output features of the moving window multi-head attention module, Q s , K s and V s Denote the linear mapping of query, key, and value matrix values ​​respectively, D denotes the dimension of query / key, and T denotes transpose.

[0108] 4.2 Angle Feature Extractor

[0109] like Figure 2 As shown in (d), the angle feature extractor AFE includes Na cascaded angle convolution units, and the angle convolution unit includes a 3×3 convolution layer and a Leaky ReLU activation function layer; wherein the output feature of the spatial feature extractor is Reshape into macropixel image stack array As the input of the first angle convolution unit, the output of the Nath angle convolution unit is used as the overall output of the angle feature extractor. The light field image is transformed into a stacked array of macro-pixel images (MacPI) in the form of Figure 3 (b) as shown.

[0110] The processing of the angle feature extractor can be expressed by the following formula:

[0111]

[0112] In the formula, represents the output of the angle feature extractor, A j represents the jth 3×3 angle convolution layer, Characterizes a macropixel image stack array.

[0113] 4.3. Epipolar Plane Image Feature Extractor

[0114] In the bottom branch of the multidimensional feature extraction module, the horizontal epipolar plane image feature extractor and the vertical epipolar plane image feature extractor are both epipolar plane image feature extractor architectures. The epipolar plane image feature extractor EPI consists of epipolar lines of different slopes, which not only span the spatial and angular dimensions, but also highlight the differences between adjacent perspectives, namely, parallax. Therefore, EPI effectively reflects the non-local correlation of space-angle and is conducive to the in-depth understanding and analysis of LF. However, due to the locality of the convolution operation, it is difficult to effectively model the non-local correlation of space-angle by processing EPI through a single convolution cascade. In order to solve this problem, the present invention constructs an EPI feature extractor to achieve robust EPI feature extraction. By using different filling rates and dilation rates, EFE has receptive fields of different scales, which can effectively capture long-distance space-angle information.

[0115] like Figure 2 As shown in (f), the epipolar plane image feature extractor includes three dilated convolution units and a convolution activation unit connected in parallel, wherein the parallel connection result of the dilated convolution units is cascaded with the convolution activation unit;

[0116] The three dilated convolution units connected in parallel are respectively a first dilated convolution unit, a second dilated convolution unit and a third dilated convolution unit; the first dilated convolution unit includes a 3×3 convolution layer with a dilation rate of 1 and a Leaky ReLU activation function layer, the second dilated convolution unit includes a 3×3 convolution layer with a dilation rate of 2, and the third dilated convolution unit includes a 3×3 convolution layer with a dilation rate of 4. A smaller dilation rate of 1 can provide finer-grained features, while a larger dilation rate, such as a dilation rate of 2, 4, etc., can increase the receptive field, and by combining convolution layers with different dilation rates, features of different scales can be captured.

[0117] The present invention transforms shallow features of a light field image into an epipolar plane image stacking array, which are respectively used as inputs of three parallel dilated convolution units, and splices the outputs of the three parallel dilated convolution units as inputs of a convolution activation unit. The output of the convolution activation unit is added to the shallow features of the input light field image as the output of an epipolar plane image feature extractor.

[0118] Specifically, in the bottom branch, the shallow features of the light field image are first Reshape into a horizontal epipolar plane image stack array As the input of the horizontal epipolar plane image feature extractor, the output obtains the horizontal epipolar plane image feature, and then the output horizontal epipolar plane image feature is Reshape into a vertical epipolar plane image stack array As the input of the vertical epipolar plane image feature extractor, the epipolar plane image feature map is extracted and used as the output of the bottom branch of the multi-dimensional feature extraction network. The light field image is transformed into a stacked array of horizontal epipolar plane images (Horizontal Epipolar-Plane Image, H-EPI) and a stacked array of vertical epipolar plane images (Vertical Epipolar-Plane Image, V-EPI) as shown in the following table. Figure 3 (c) Figure 3 (d) as shown.

[0119] The processing of the horizontal epipolar plane image feature extractor and the vertical epipolar plane image feature extractor can be expressed as:

[0120]

[0121] In the formula, and denote the outputs of the horizontal epipolar plane image feature extractor and the vertical epipolar plane image feature extractor respectively; Characterizes the stacked array of horizontal epipolar plane images, represents the features of the stacked array of vertical epipolar plane images, and E represents the epipolar plane image feature extractor.

[0122] 5. Upsampling Reconstruction Network

[0123] like Figure 2 As shown in (a), the upsampling reconstruction network (Feature Reconstruction) upsamples the 4D light field features of the light field image, and then adds them to the feature map obtained by the bicubic interpolation method of the light field image input to the model to generate a spatial super-resolution reconstructed image of the light field image as the model output.

[0124] like Figure 2 As shown in (g), the upsampling reconstruction network includes a first 1×1 convolutional layer, a pixelshuffle layer, and a second 1×1 convolutional layer cascaded in sequence; wherein the input of the upsampling reconstruction network is used as the input of the first 1×1 convolutional layer, increasing the number of input feature channels to α 2C. Then, the output of the first 1×1 convolutional layer is used as the input of the pixelshuffle layer to reshape the output features of the first 1×1 convolutional layer from the channel dimension to the spatial dimension to generate super-resolution features αH×αW. Finally, the super-resolution features αH×αW are used as the input of the second 1×1 convolutional layer for super-resolution feature refinement, and the output is the output of the upsampling reconstruction network; α is the spatial resolution magnification factor.

[0125] It should be noted that the method of the present invention only performs super-resolution on the Y channel of the LF image, while the Cb and Cr channels are reconstructed using the bicubic interpolation method.

[0126] 6. Training of Spatial Super-resolution Reconstruction Model

[0127] In a specific implementation, the spatial super-resolution reconstruction model is trained in the following manner: the divided training set is used as the input of the spatial super-resolution reconstruction model, a loss function is constructed that is robust to discrete values, and the model image of the spatial super-resolution reconstruction model is reconstructed with the goal of minimizing the loss function, thereby training the spatial super-resolution reconstruction model.

[0128] In the application of the present invention, the training loss is preferably calculated by the following formula:

[0129]

[0130] Where, L 1 represents the loss function, Represents the low-resolution light field image of the mth training sample The spatial super-resolution reconstructed image, represents the training set, Represents the low-resolution light field image of the mth training sample The corresponding high-resolution light field image, M represents the number of training samples contained in the training sample set.

[0131] VII. Experimental Description

[0132] In order to better illustrate the advantages of the technical solution of the present invention, this embodiment discloses the following experiment.

[0133] This experimental example selected 167 LF (light field image) images from five public databases: HCI_new, HCI_old, INRIA_Lytro, Stanford_Gantry, and EPFL. Among them, 144 scenes were used for training and 23 scenes were used for testing. The angular resolution of all datasets is 9×9. This experimental example only extracted the central 5×5 LF for training and testing. In the training stage, this experimental example extracted image blocks of size 64×64 / 128×128 from each LF, and used the bicubic interpolation method to perform 0.5 / 0.25 downsampling to generate 2× / 4×LR training samples. In addition, data augmentation techniques such as horizontal and vertical flipping and 90-degree rotation were applied to enhance the generalization ability of the data. In the test stage, the LF images were cropped into image blocks, and SR (super-resolution reconstruction) was performed separately on these image blocks. Subsequently, the SR image blocks were aggregated to generate high-resolution LF images. In order to quantitatively evaluate the reconstruction accuracy, this experimental example uses Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) as evaluation indicators.

[0134] In this experimental example, the number of channels C is set to 64. The window of the Swin Transformer is 16×16 and the moving window is 8×8. In addition, N, Ns, and Na are set to 6, 6, and 1, respectively. In all experiments, the Adam optimizer is used for training, where β1=0.99 and β2=0.999. The convolution kernel weights are initialized using the Xavier algorithm. During the training phase, the batch size is 4 and the initial learning rate is 2×10 -4 The learning rate was halved every 15 epochs, and the training lasted for a total of 50 epochs. The network in this experimental example was implemented in the Pytorch framework and trained on a PC equipped with a single NVIDA RTX 4090 GPU.

[0135] To evaluate the performance of our method, we compared it with some state-of-the-art methods on five public datasets. Two single image methods and 13 LF space SR methods were included. The bicubic interpolation method was used as the baseline. All methods were tested under the same experimental conditions for fair comparison.

[0136] Table 1 Comparison of PSNR(dB) / SSIM / LPIPS of different methods in 2× / 4× tasks.

[0137]

[0138] Table 1 lists the quantitative results of different methods at different scales on eight datasets, with the best result marked in red and the second best result marked in blue. The proposed method achieves excellent average quantitative scores in the 2×SR task. Compared with EDSR and RCAN, the proposed method achieves an average PSNR gain of 3.09dB and 2.93dB, respectively. This is mainly because EDSR and RCAN process each SAI independently and ignore the correlation between views. Therefore, the reconstruction performance is limited. In addition, compared with resLF, the proposed method achieves an average PSNR gain of 1.63dB. This shows that using only partial view information leads to poor reconstruction quality. Although MEG-Net further optimizes resLF through 3D convolution, the performance gain is still limited. Compared with two early decoupling mechanism methods, LFSSR and LF-InterNet, the proposed method achieves an average PSNR gain of 1.63dB and 1.11dB, respectively. LFSSR and LF-InterNet fail to exploit the rich spatial and angular information in EPI, which limits the reconstruction performance. The DistgSSR and LFSSR-SAV methods effectively integrate the feature information of the three subspace images and achieve advanced performance on the 2×SR task. In contrast, the proposed method is still superior to them. This shows that the proposed method is able to better extract feature information from the three subspace images. Compared with the two Transformer methods DPT and LFT, the proposed method still achieves higher reconstruction performance, with average PSNR improved by 0.73dB and 0.38dB, respectively. In terms of SSIM and LPIPS, the proposed method also achieves excellent results. This shows that the proposed method is able to reconstruct images that are more in line with human visual expectations.

[0139] In the 4×SR task, the average PSNR values ​​of our method on small parallax scenes (EPFL and INRIA) are 0.65dB and 0.75dB lower than those of LFGN, respectively. However, on the HCInew, HCInold, and STFgantry datasets, the proposed method shows more promising results. Since LFGN utilizes a feature modulation-based CNN model for LF modeling, it achieves reliable gains in local feature extraction. In contrast, the proposed method balances the performance over various disparity ranges, which may result in limited local modeling capabilities. In addition, the results in Table 1 show that the proposed method also exhibits advanced performance in various disparity range scenes compared to other existing advanced methods.

[0140] Figure 4 and Figure 5The SR results of different methods for the real scene EPFL_ISO_Chart_1 and the synthetic scene HCInew_Bedroom under 2× and 4× SR tasks are presented respectively. In each figure, the local magnified areas of SAI and EPI are framed by red and cyan rectangles, respectively. For the convenience of subjective comparison, red arrows are used to highlight the specific details. For the reconstruction quality of the scene EPFL_ISO_Chart_1 under 2×SR, EDSR and RCAN achieve higher PSNR and SSIM compared with the existing methods of LF spatial SR methods. However, the reconstructed images show obvious artifacts. Although EDSR and RCAN can capture a larger receptive field and utilize more pixels within the view for reconstruction, they process each SAI independently. Therefore, they ignore the correlation between different views and cannot effectively process inter-view information. The LF spatial SR method based on deep learning can simultaneously utilize the spatial and angular information of LF, thereby better reconstructing clear texture details. However, in the scene EPFL_ISO_Chart_1 reconstructed by DPT, the wedge-shaped area shows false bending artifacts (see the corresponding position indicated by the red arrow). Other LF SR methods and the proposed method can reconstruct the details in SAI better. In addition, the proposed method presents most of the fine-grained straight lines in EPI. This shows that the proposed method can effectively preserve the disparity structure of LF.

[0141] Due to the stronger ill-posedness of 4×SR, the reconstruction quality of each method is degraded. The single-image methods EDSR and RCAN cannot reliably recover the lost texture details, resulting in obvious artifacts in the reconstructed image. Although the existing LF image SR methods effectively improve the reconstruction quality, Figure 5 The position indicated by the red arrow in the figure is still blurred. In contrast, the proposed method is able to generate finer texture details in this area. At the same time, the EPI of HCInew_Bedroom restored by the proposed method shows more consistent straight lines. This further shows that the proposed method can better preserve the disparity of LF. In addition, the proposed method achieves the highest PSNR and SSIM quantitative scores in both 2× and 4× SR tasks for both scenes.

[0142] By analyzing the reconstruction results, the method of the present invention achieves a higher reconstruction performance, which is mainly due to the fact that it fully utilizes the feature information of the three 2D subspace images, thereby maximally restoring the fine texture in SAI and the epipolar details in EPI.

[0143] This experimental example further analyzes the performance of the existing state-of-the-art methods and the proposed method in terms of angular consistency of reconstruction quality. Figure 6The PSNR distribution of scenes EPFL_Bikes and INRIA_Building in the 2×SR task is shown. This experimental example provides the average PSNR and mean square deviation (Std) between views. Compared with the existing state-of-the-art methods, the proposed method achieves relatively high average PSNR and low Std values. This means that the proposed method can effectively ensure the angle consistency of LF while achieving higher reconstruction quality.

[0144] Table 2 Comparison of parameters, floating-point operations, average running time, and average PSNR(dB) / SSIM / LPIPS of different methods in 2×SR.

[0145]

[0146] This experimental example further uses the number of parameters, floating-point operations (FLOPs), average running time, PSNR, SSIM, and LPIPS to evaluate the reconstruction efficiency of the proposed method. The average running time is the time to reconstruct a single SAI image. All experiments use input image blocks of size 5×5×32×32 and are accelerated using the NVIDIA RTX 4090 GPU image processor in the Pytorch environment. The detailed experimental results are shown in Table 2, with the best result marked in red and the second best result marked in blue. LFSSR has the smallest parameters, running time, and FLOPs, but the reconstruction performance is relatively limited. Compared with existing state-of-the-art methods such as DistgSSR, LFSSR-SAV, LF-DGNet, DPT, and LFT, the proposed method has relatively high parameters and FLOPs, but performs best in terms of reconstruction quality. Due to the inherent limitations of memory access patterns, Transformer-based methods run slower than CNN-based methods. Benefiting from the window-based self-attention mechanism, the proposed method surpasses LFT and DPT in terms of running time. In addition, it also achieves higher reconstruction accuracy. In general, the method of the present invention achieves a good balance between reconstruction accuracy and computational efficiency.

[0147] Depth estimation is one of the important applications of LF. In order to explore whether the proposed method can be well applied to depth estimation applications, this experimental example performs depth estimation on the scene HCIold_buddha 4×SR results. Figure 7 As shown in Figure 1, the depth map of the original HRLF image is used as the baseline. The root mean square error (RMSE) multiplied by 100 is used as a quantitative measurement indicator. Figure 5From the information, we can see that the single image method RCAN achieves better results than the bicubic method in terms of PSNR and SSIM. However, RCAN achieves poor results in depth estimation. The root cause is that RCAN processes each SAI independently during the SR process without utilizing the angle information, resulting in a large amount of error information in the depth map. In contrast, the existing LF space SR methods generate depth maps with fewer error pixels and higher quantitative scores. In addition, the RMSE of this method is very close to that of the original HR LF. This means that this method achieves high reconstruction quality while effectively maintaining the LF angle consistency. Therefore, it is very suitable for depth estimation applications.

[0148] LF refocusing is a technique that uses LF data for image processing and editing, which can change the focus position of the image. SR technology can improve the detail texture of the refocused LF image. In order to verify the effectiveness of the proposed method in LF refocusing, this experimental example uses the LF refocusing method and the 2×SR method to obtain images of three different focal planes of the scene Tarot_Cards_S. Figure 8 As shown in Figure 1, the refocused image area generated by the proposed method is very close to the area of ​​the original HR image. This experimental example also uses the PSNR indicator to evaluate the quality of the refocused image. The results show that the refocused area generated by the proposed method has a high PSNR value. This further proves the effectiveness of the proposed method in low-frequency refocusing applications.

[0149] Table 3 PSNR(dB) / SSIM / LPIPS of different 2×SR models and the baseline model.

[0150]

[0151] The spatial and angular dimensions are asymmetric. Therefore, the present invention proposes an asymmetric method to process SAI and MacPI, aiming to extract more intrinsic information from LF. In order to verify the effectiveness of this asymmetric processing, several models are introduced in this experimental example: Model-1, Model-2, Model-3, Model-4, Model-5, Model-6, and Model-7. It is worth noting that Model-1, Model-2, Model-3, Model-4, and Model-5 are constructed by reducing the spatial convolution layer of SFE in Model-Base. At the same time, Model 6 and Model 7 correspond to the models in which the angular convolution layer of AFE is added to Model-Base.

[0152] As shown in Table 3, as the number of SFE convolutional layers decreases, the reconstruction performance gradually decreases. In Table 3, the best result is marked in red and the second best result is marked in blue. Among them, Model-1 has the worst performance. In particular, on the STFgantry dataset, the PSNR of Model-1 is 1.29dB lower than that of Model-Base. This shows that the enhanced SAI processing helps to improve the overall quality of the reconstructed image. On the other hand, although Model-6 and Model-7 have stronger angular feature processing capabilities than Model-base, they do not achieve the expected improvement in reconstruction quality. On the contrary, the performance of both models is worse than that of Model-Base. This shows that the additional angular feature processing is redundant.

[0153] In summary, asymmetric processing in spatial and angular dimensions can enhance the representation ability of spatial information and reduce the redundancy of angular information.

[0154] As mentioned above, it is difficult to effectively model the spatial-angular non-local correlation using a single convolution cascade to process EPI. Therefore, the present invention introduces EFE to process EPI, aiming to expand the receptive field range in the spatial and angular dimensions. In order to evaluate the effectiveness of EFE, this experimental example constructs two models, ModelBase-W-A1 and ModelBase-A2. In ModelBase-W-A1, this experimental example uses residual convolution instead of EFE, and in ModelBase-W-A2, this experimental example uses conventional convolution instead of the dilated convolution in EFE. As shown in Table 3, the SR performance of ModelBase-W-A1 and ModelBase-W-A2 is not as good as that of Model-Base. This shows that by using different padding and dilation rates, the spatial-angular information in EPI can be deeply extracted.

[0155] SFE combines the efficient local feature extraction capability of dense convolutional networks and the powerful global modeling capability of Swin Transformer to handle SAI. To verify the contribution of these two components, this experimental example introduces two models: Model-Base-WS and Model-Base-WD. Among them, Model-Base-WS removes SwinTransformer from Model-Base, while Model-Base-WD removes dense skip connections from Model-Base. As shown in Table 3, dense skip connections have a significant impact on SR performance. For example, the PSNR of ModelBase-WD on the STFgantry dataset drops by 1.22dB. This shows that a single convolution cascade is not enough to extract rich spatial information. In addition, the SR performance of Model-Base-WS is worse than that of Model-Base. This shows that more comprehensive spatial information can be extracted using Swin Transformer.

[0156] 8. Overview

[0157] In summary, the method of the present invention has the following technical advantages:

[0158] (1) In the method of the present invention, a spatial feature extractor and an angular feature extractor are constructed respectively, and different receptive fields are used to asymmetrically extract spatial and angular features, thereby maximizing the utilization of feature information in sub-aperture images and macro-pixel images. This solves the problem that the existing light field image spatial super-resolution method ignores the spatial-angular asymmetry, resulting in insufficient feature information extraction in sub-aperture images and macro-pixel images.

[0159] (2) In the method of the present invention, an epipolar plane image feature extractor is constructed. By using different filling rates and dilation rates, the epipolar plane feature extractor has receptive fields of different scales, thereby being able to effectively capture long-distance spatial-angular information and achieve effective modeling of spatial-angular non-local correlations. This solves the problem that the existing light field image spatial super-resolution method uses conventional convolution to extract feature information of the epipolar plane image, ignores the spatial-angular non-local correlation, and fails to fully utilize the long-distance spatial-angular feature information.

[0160] (3) The super-resolution reconstruction results of light field images obtained by the method proposed in the present invention are superior to similar methods in the prior art in terms of visual quality and quantitative quality indicators, providing a more effective technical auxiliary means for the wider practical application of light field images.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A light field image spatial super-resolution method based on a space-angle decoupling mechanism, characterized in that: include: S1: Acquire the target light field image to be reconstructed; S2: input the target light field image into the trained spatial super-resolution reconstruction model, and output a spatial super-resolution reconstructed image of the target light field image; The training steps of the spatial super-resolution reconstruction model are as follows: S201: Converting a high-resolution light field image into a low-resolution light field image as a training sample, and inputting the result into a spatial super-resolution reconstruction model; the spatial super-resolution reconstruction model includes a shallow feature extraction network, a multi-dimensional feature extraction network, and an upsampling reconstruction network; S202: The shallow feature extraction network maps the light field image input by the model into shallow features as an input feature map of the multidimensional feature extraction network; S203: The multidimensional feature extraction network includes a spatial feature extractor, an angular feature extractor and an epipolar plane image feature extractor, which respectively extract spatial features, angular features and epipolar plane image features from the subspace image, macropixel image and epipolar plane image of the input feature map, and fuse them to generate 4D light field features of the light field image as input of the upsampling reconstruction network; S204: the upsampling reconstruction network performs upsampling and reconstruction processing on the 4D light field features of the light field image to generate a spatial super-resolution reconstructed image of the light field image; S205: calculating a training loss according to the spatial super-resolution reconstructed image of the light field image as a training sample and its corresponding high-resolution light field image, and optimizing parameters of the spatial super-resolution reconstruction model based on the training loss; S206: Repeat steps S201 to S205 until the spatial super-resolution reconstruction model converges or reaches a preset number of iterations; S3: The spatial super-resolution reconstructed image output by the spatial super-resolution reconstruction model is used as the super-resolution reconstruction result of the target light field image.

2. The light field image spatial super-resolution method based on the space-angle decoupling mechanism according to claim 1 is characterized in that: In step S201, the high-resolution light field image is cropped, and the cropped light field image is downsampled using a bicubic interpolation method to obtain a low-resolution light field image as a training sample.

3. The light field image spatial super-resolution method based on the space-angle decoupling mechanism according to claim 1, characterized in that: In step S203, the multidimensional feature extraction network includes a plurality of multidimensional feature extraction modules cascaded in sequence, each of which includes a top branch, a bottom branch and a connection layer; The top branch includes a cascaded spatial feature extractor and an angular feature extractor. The spatial feature extractor is used to extract the spatial features of the sub-aperture image in the input feature map. The obtained spatial feature map is input to the angular feature extractor to further extract the angular features of its macro-pixel image, thereby obtaining a multi-scale spatial-angular feature map of the input feature map. The bottom branch includes a horizontal epipolar plane image feature extractor and a vertical epipolar plane image feature extractor which are cascaded in sequence by weight sharing, the horizontal epipolar plane image feature extractor is used to extract horizontal epipolar plane image features of the input feature map, the obtained horizontal epipolar plane image feature map is input to the vertical epipolar plane image feature extractor for further extracting vertical epipolar plane image features, and obtaining an epipolar plane image feature map of the input feature map; The connection layer is used to splice the obtained multi-scale spatial-angular feature map and the epipolar plane image feature map, and after activation processing through the convolution activation unit, it is fused with the input feature map to generate the 4D light field features of the light field image as the input of the upsampling reconstruction network.

4. The light field image spatial super-resolution method based on the space-angle decoupling mechanism according to claim 3 is characterized in that: The spatial feature extractor includes a dense convolutional network, a convolutional activation unit and a SwinTransformer module connected in sequence; The dense convolution network includes Ns cascaded spatial convolution units, and the spatial convolution unit includes a cascaded 3×3 convolution layer and a leaky ReLU activation function layer; wherein the input of the spatial feature extractor is respectively used as the input of the Ns spatial convolution units, and the output of the previous spatial convolution unit is respectively used as the input of the subsequent spatial convolution unit, and the output of the Ns spatial convolution units is concatenated as the output of the dense convolution network, and the output of the dense convolution network is used as the input of the convolution activation unit; The convolution activation unit includes a cascaded 1×1 convolution layer and a Leaky ReLU activation function layer; The Swin Transformer module includes a cascaded layer-normalized multi-head attention unit and a layer-normalized perception unit, wherein the layer-normalized multi-head attention unit includes a cascaded layer-normalized layer and a moving window multi-head self-attention module, and the layer-normalized perception unit includes a cascaded layer-normalized layer and a multi-layer perceptron; wherein the input of the Swin Transformer module is used as the input of the layer-normalized multi-head attention unit, the output of the layer-normalized multi-head attention unit is added to the input of the Swin Transformer module as the input of the layer-normalized perception unit, and the output of the layer-normalized perception unit is added to the input of the layer-normalized perception unit as the output of the Swin Transformer module; In the spatial feature extractor, the shallow feature shaping of the light field image is first converted into a stacked array of sub-aperture images as the input of the dense convolutional network. The output of the dense convolutional network is used as the input of the convolutional activation unit to generate a multi-scale spatial feature map as the output of the convolutional activation unit. The multi-scale spatial feature map is used as the input of the Swin Transformer module, and the output of the Swin Transformer module is used as the overall output of the spatial feature extractor.

5. The light field image spatial super-resolution method based on the space-angle decoupling mechanism according to claim 4 is characterized in that: The processing of the spatial feature extractor is expressed by the following formula: F LN =LN(F D ); F S(W) =SW-MSA(F LN )+F D ; In the formula, S i represents the i-th densely connected 3×3 spatial convolutional layer, Conv 1×1 represents a 1×1 convolutional layer, cat represents channel cascade, [·] represents cascade, and F D Represents dense convolutional networks and Conv 1×1 The generated multi-scale spatial features, represents the characteristics of the subaperture image stack array, F LN Represents the output of the normalized layer in the multi-head attention unit, F S(W) The input of the normalized perception unit of the representation layer, Represents the output of the spatial feature extractor; SW-MSA represents the moving window multi-head self-attention module, MLP represents the multi-layer perceptron, and LN represents the regularization processing of the layer normalization layer.

6. The light field image spatial super-resolution method based on the space-angle decoupling mechanism according to claim 3, characterized in that: The angle feature extractor includes Na cascaded angle convolution units, and the angle convolution unit includes a 3×3 angle convolution layer and a Leaky ReLU activation function layer; In the angle feature extractor, the output feature of the spatial feature extractor is shaped and converted into a macro-pixel image stack array as the input of the first angle convolution unit, and the output of the Nath angle convolution unit is used as the overall output of the angle feature extractor; the processing process of the angle feature extractor is expressed by the following formula: In the formula, represents the output of the angle feature extractor, A j represents the jth 3×3 angle convolution layer, Characterizes a macropixel image stack array.

7. The light field image spatial super-resolution method based on space-angle decoupling mechanism according to claim 3, characterized in that: The horizontal epipolar plane image feature extractor and the vertical epipolar plane image feature extractor in the bottom branch are both epipolar plane image feature extractors, and the epipolar plane image feature extractors include three dilated convolution units and a convolution activation unit connected in parallel, wherein the parallel connection result of the dilated convolution units is cascaded with the convolution activation unit; The three dilated convolution units connected in parallel are respectively a first dilated convolution unit, a second dilated convolution unit and a third dilated convolution unit; the first dilated convolution unit includes a 3×3 convolution layer with a dilation rate of 1 and a Leaky ReLU activation function layer, the second dilated convolution unit includes a 3×3 convolution layer with a dilation rate of 2, and the third dilated convolution unit includes a 3×3 convolution layer with a dilation rate of 4; the outputs of the three dilated convolution units connected in parallel are spliced ​​and used as the input of the convolution activation unit, and the output of the convolution activation unit is then added to the shallow features of the input light field image as the output of the epipolar plane image feature extractor; In the bottom branch, the shallow features of the light field image are first shaped and converted into a stacked array of horizontal epipolar plane images as the input of the horizontal epipolar plane image feature extractor, and the horizontal epipolar plane image features are extracted. Then, the output horizontal epipolar plane image features are shaped and converted into a stacked array of vertical epipolar plane images as the input of the vertical epipolar plane image feature extractor, and the epipolar plane image feature map is extracted as the output of the bottom branch in the multidimensional feature extraction network.

8. The light field image spatial super-resolution method based on space-angle decoupling mechanism according to claim 3, characterized in that: The processing of the horizontal epipolar plane image feature extractor and the vertical epipolar plane image feature extractor are respectively expressed as follows: In the formula, and denote the outputs of the horizontal epipolar plane image feature extractor and the vertical epipolar plane image feature extractor respectively; Characterizes the stacked array of horizontal epipolar plane images, represents the features of the stacked array of vertical epipolar plane images, and E represents the epipolar plane image feature extractor.

9. The light field image spatial super-resolution method based on space-angle decoupling mechanism according to claim 1, characterized in that: In step S204, the upsampling reconstruction network upsamples the 4D light field features of the light field image, and then adds the upsampling network to the feature map obtained by bicubic interpolation of the light field image input by the model to generate a spatial super-resolution reconstructed image of the light field image output by the model; The upsampling reconstruction network includes a first 1×1 convolutional layer, a pixelshuffle layer, and a second 1×1 convolutional layer cascaded in sequence; wherein the input of the upsampling reconstruction network is used as the input of the first 1×1 convolutional layer, and the number of input feature channels is increased to α 2 C. Then, the output of the first 1×1 convolutional layer is used as the input of the pixelshuffle layer to reshape the output features of the first 1×1 convolutional layer from the channel dimension to the spatial dimension to generate super-resolution features αH×αW. Finally, the super-resolution features αH×αW are used as the input of the second 1×1 convolutional layer for super-resolution feature refinement, and the output is the output of the upsampling reconstruction network; α is the spatial resolution magnification factor.

10. The light field image spatial super-resolution method based on space-angle decoupling mechanism according to claim 1, characterized in that: In step S205, the training loss is calculated by the following formula: In the formula, L1 represents the loss function, Represents the low-resolution light field image of the mth training sample The spatial super-resolution reconstructed image, Represents the low-resolution light field image of the mth training sample The corresponding high-resolution light field image, M represents the number of training samples contained in the training sample set.

Images