Light field image spatial super-resolution method based on multi-scale large kernel attention mechanism

Abstract

The invention relates to the technical field of light field image super-resolution reconstruction, in particular to a light field image spatial super-resolution method based on a multi-scale large kernel attention mechanism, which comprises the following steps: inputting a target light field image into a trained spatial super-resolution reconstruction model, and outputting a spatial super-resolution reconstruction image of the target light field image; in the spatial super-resolution reconstruction model, performing feature extraction on an input light field image through a shallow feature extraction network to generate a shallow feature map; respectively extracting angle features, spatial features and epipolar plane image features from the macro-pixel image of the shallow feature map, the sub-aperture image for generating the macro-pixel feature map and the epipolar plane image of the shallow feature map through a large kernel feature extraction network, and fusing to generate a 4D light field feature map; and performing up-sampling reconstruction on the 4D light field feature map through an up-sampling reconstruction network to generate a spatial super-resolution reconstruction image of the input light field image. According to the invention, the reconstruction quality of the spatial super-resolution of the light field image can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of light field image super-resolution reconstruction, and particularly relates to a light field image spatial super-resolution method based on a multi-scale large kernel attention mechanism. BACKGROUND

[0002] Light field (LF) imaging can capture both intensity and directional information of light rays simultaneously. This capability makes it widely used in various vision tasks, such as object detection, depth estimation, digital refocusing, and three-dimensional (3D) reconstruction. However, due to the resolution limitation of light field imaging sensors, there is a trade-off between spatial resolution and angular resolution for light field images. Currently, the spatial resolution of light field captured by commercial light field cameras is limited, which restricts the application of light field images in many fields. Therefore, it is of great practical significance to develop a robust and efficient light field image spatial super-resolution (SR) method.

[0003] By Figure 2 It can be seen that the sub-aperture images (SAIs) preserve the strong local spatial correlation of natural images, and the convolutional neural network (CNN) is very suitable for extracting spatial details due to its translational invariance and ability to capture local textures. The representation of macro-pixel images (MacPIs) presents a highly coupled spatial-angular distribution and mosaic texture. Given the non-local long-range dependency in MacPIs, the Transformer can effectively decouple and aggregate global context information through the self-attention mechanism. The significant linear structure in the epipolar plane image (EPI) directly reflects the disparity clue. CNN can accurately capture the slope and geometric continuity of these lines, thereby ensuring the geometric consistency of the light field.

[0004] Although the light field has the above advantages in the light field image space super-resolution task, efficiently and cooperatively capturing the angle low-frequency information and the spatial high-frequency information of the light field image is still a problem for improving the reconstruction performance. The traditional light field image space super-resolution method mainly uses manually extracted image priors to improve the resolution, and the quality of the reconstructed image is poor. With the rapid development of deep neural networks, many learning-based methods have been applied to the field of light field image space super-resolution. The existing methods mainly rely on CNNs sensitive to spatial high-frequency information to extract features from three typical light field visualization representations, namely SAIs, MacPIs and EPIs. However, this kind of method is limited by the inherent receptive field limitation of convolution, and it is difficult to effectively model the long-range spatial-angle correlation features existing in the light field image. Although a large number of methods introduce the Transformer architecture with long-distance dependence modeling capability for the light field super-resolution task, the disadvantage of the Transformer in capturing local detail features has not been effectively compensated, which directly leads to the fact that the reconstructed result still has room for improvement in detail fidelity. To solve the problems caused by CNN and Transformer, some methods combine CNN and Transformer, which greatly improves the super-reconstruction performance. However, these methods still use ordinary convolution with limited receptive field for feature extraction, so there is still room for further improvement in reconstruction quality.

[0005] Therefore, how to efficiently extract the spatial high-frequency information and the angle low-frequency information of the light field image has become a problem that technicians in the field need to solve. SUMMARY

[0006] In view of the above problems of the prior art, the technical problem to be solved by the present application is to provide a light field image space super-resolution method based on a multi-scale large kernel attention mechanism, a hybrid CNN-Transformer architecture is constructed, the advantages and disadvantages of CNN and Transformer are used to fully obtain feature information in sub-aperture images, macro-pixel images and epipolar plane images, and a large kernel attention module is used to replace the convolution feature extraction part to increase the receptive field of feature extraction, and deep features of the light field are obtained, thereby improving the reconstruction quality of the light field image space super-resolution.

[0007] In order to solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0008] The light field image space super-resolution method based on the multi-scale large kernel attention mechanism comprises:

[0009] S1: obtaining a target light field image to be reconstructed;

[0010] S2: Input the target light field image into the trained spatial super-resolution reconstruction model, and output the spatial super-resolution reconstructed image of the target light field image;

[0011] The processing steps during training of a spatial super-resolution reconstruction model include:

[0012] S201: Convert high-resolution light field images into low-resolution light field images and use them as training samples to input into the spatial super-resolution reconstruction model;

[0013] S202: Extract features from the input light field image using a shallow feature extraction network to generate a shallow feature map;

[0014] S203: An angular feature, spatial feature, and epipolar plane image feature are extracted from the macro-pixel image of the shallow feature map, the sub-aperture image that generates the macro-pixel feature map, and the epipolar plane image of the shallow feature map, respectively, and then fused to generate a 4D light field feature map.

[0015] S204: Upsample and reconstruct the 4D light field feature map through an upsampling reconstruction network to generate a spatial super-resolution reconstructed image of the input light field image;

[0016] S205: Calculate the training loss based on the spatial super-resolution reconstructed image of the input light field image and the corresponding high-resolution light field image, and optimize the parameters of the spatial super-resolution reconstruction model based on the training loss;

[0017] S206: Repeat steps S201 to S205 to iteratively train the spatial super-resolution reconstruction model until the model converges or reaches the preset number of iterations.

[0018] S3: Use the spatial super-resolution reconstructed image output by the spatial super-resolution reconstruction model as the super-resolution reconstruction result of the target light field image.

[0019] Preferably, in step S203, the processing steps of the large kernel feature extraction network include:

[0020] S2031: Reshape the input shallow feature map into a macro-pixel image, and input it into two cascaded angle Transformers for angle feature extraction to generate a low-frequency angle feature map;

[0021] S2032: Reshape the low-frequency feature map of the angle into a sub-aperture image and input it into the dense large kernel attention module for spatial feature extraction of the high-frequency feature map of the space;

[0022] S2033: After inputting the high-frequency spatial feature map into a 1×1 convolutional layer for dimensionality reduction, it is reshaped into a spatial-angular feature map with the same tensor as the input shallow feature map.

[0023] S2034: Reshape the input shallow feature map into a horizontal epipolar image, and input it into two cascaded large-kernel attention units for epipolar plane image feature extraction to generate a horizontal epipolar feature map;

[0024] S2035: Reshape the horizontal epipolar feature map into a vertical epipolar image, and input it into two cascaded large-kernel attention units for epipolar plane image feature extraction to generate a vertical epipolar feature map;

[0025] S2036: Reshape the vertical epipolar feature map into an epipolar planar image feature map with the same tensor as the input shallow feature map;

[0026] S2037: The spatial-angle feature map and the epipolar plane image feature map are concatenated and fed into a 1×1 convolutional layer for dimensionality reduction. Then, they are added element-wise to the input shallow feature map to generate the 4D light field feature map output by the large kernel feature extraction network.

[0027] Preferably, in step S2031, the angle Transformer includes a cascaded first normalization layer, a multi-head self-attention layer, a second normalization layer, and a multilayer perceptron.

[0028] The input of the Angle Transformer is added element-wise to the output of the multi-head self-attention to generate the input of the second normalized layer; the input of the second normalized layer is added element-wise to the output of the multilayer perceptron to generate the output feature of the Angle Transformer.

[0029] Preferably, in step S2032, the dense large-kernel attention module includes several cascaded large-kernel attention units;

[0030] For each large kernel attention unit, its output is concatenated with the outputs of all preceding large kernel attention units and the input of the dense large kernel attention module to generate the input of the next large kernel attention unit; the output of the last large kernel attention unit is concatenated with the outputs of all preceding large kernel attention units and the input of the dense large kernel attention module to generate the output feature map of the dense large kernel attention module.

[0031] Preferably, the large kernel attention unit includes a cascaded first normalization layer, a multi-branch kernel decomposition attention module, a second normalization layer, and a bi-branch convolutional gating module;

[0032] The input of the large kernel attention unit is added element-wise to the output of the multi-branch kernel decomposition attention module to generate the input of the second normalized layer; the input of the second normalized layer is added element-wise to the output of the bi-branch convolutional gating module to generate the output feature of the large kernel attention unit.

[0033] Preferably, the processing steps of the multi-branch kernel decomposition attention module include:

[0034] 1) Input the input feature map into a 1×1 pointwise convolutional layer for feature extraction and dimensionality upscaling to generate an initial feature map;

[0035] 2) Perform channel segmentation on the initial feature map to obtain the first segmentation feature map and the second segmentation feature map;

[0036] 3) Input the first segmentation feature map into the four parallel branches for attention enhancement, and then concatenate the outputs of the four branches to generate a fused feature map;

[0037] Each branch includes cascaded depthwise convolutional layers, depthwise dilated convolutional layers, and pointwise convolutional layers. The output feature map of each branch is generated by multiplying the output of the depthwise convolutional layer and the output of the pointwise convolutional layer element-wise.

[0038] 4) After the second segmentation feature map is input into the pointwise convolutional layer for processing, it is multiplied elementwise with the fused feature map to generate the output feature map of the multi-branch kernel decomposition attention module.

[0039] Preferably, the dual-branch convolutional gating module includes a feature branch and a gating branch;

[0040] Feature branches include pointwise convolutional layers;

[0041] Gated branches include cascaded pointwise convolutional layers and depthwise convolutional layers;

[0042] After the outputs of the first branch and the second branch are multiplied element-wise, they are input into the pointwise convolutional layer for processing to generate the output feature map of the dual-branch convolutional gating module.

[0043] Preferably, the processing steps of the dual-branch convolutional gating module include:

[0044] 1) Input the input feature map into the feature branch, and extract the initial features through a pointwise convolutional layer to obtain the feature map. ;

[0045] 2) Input the feature map into the gated branch, and extract the initial features through pointwise convolutional layers to obtain the feature map. Subsequently, spatial context information is captured through depthwise convolutional layers to generate a spatial gating weight map. ;

[0046] The formula is expressed as:

[0047] ;

[0048] In the formula, DWConv 7×7 Represents 7×7 depthwise convolution

[0049] 3) Feature map Spatial Gating Weight Map Perform element-wise multiplication to generate gated modulation. ;

[0050] The formula is expressed as:

[0051] ;

[0052] In the formula: This represents element-wise multiplication;

[0053] 4) Gating modulation After cross-channel weighted aggregation of the input pointwise convolutional layer, a residual connection is formed with the input feature map, and then processed through learnable parameters. Adaptively adjust attention weight strength to generate output feature maps of the bi-branch convolutional gating module. ;

[0054] The formula is expressed as:

[0055] ;

[0056] In the formula: This indicates a pointwise convolutional layer. This represents the input to the bi-branch convolutional gating module.

[0057] Compared with existing technologies, the light field image spatial super-resolution method based on the multi-scale large kernel attention mechanism in this invention has the following advantages:

[0058] To address the unresolved issue in existing light field image spatial super-resolution methods of failing to simultaneously and efficiently extract spatial and angular information from light field images, this invention proposes a light field image spatial super-resolution method based on a multi-scale large-kernel attention mechanism. On one hand, it utilizes a Transformer with global context modeling capabilities to capture low-frequency angular information, and a CNN sensitive to local details to capture high-frequency spatial information and horizontal / vertical epipolar plane image features, making it suitable for reconstruction tasks of light field scenes with large parallax. On the other hand, it expands the receptive field range for spatial and epipolar plane feature extraction through dense connection structures and large-kernel convolution decomposition. It utilizes multi-scale characteristics to encode the light field image from coarse to fine, enabling the simultaneous extraction of global and local information from the light field image. Furthermore, it employs dual-branch convolutional gating units to enhance feature representation, achieving effective modeling of spatial-angular nonlocal correlations while maintaining a lightweight model. This solves the problem of existing light field image spatial super-resolution methods only using conventional convolution to extract feature information from epipolar plane images, thereby improving the reconstruction quality of light field image spatial super-resolution.

[0059] Experimental results on real-world and synthetic light field image datasets demonstrate that the super-resolution reconstruction results of light field images obtained by the method proposed in this invention outperform similar methods in the prior art in terms of both visual quality and quantitative quality indicators, providing a more effective technical support for the wider practical application of light field images. Attached Figure Description

[0060] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0061] Figure 1 This is a network structure diagram of a spatial super-resolution reconstruction model.

[0062] Figure 2 This is an example diagram showing the transformation of three 2D subspace images of a light field image (subaperture image, macropixel image, and epipolar plane image). Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0064] The following detailed explanation illustrates the specific implementation methods:

[0065] Example:

[0066] Light field imaging records both the intensity and angle of light, and has potential applications in many fields. However, some commercial microlens cameras sacrifice spatial resolution to achieve higher angular resolution, resulting in low spatial resolution of the acquired light field images, which limits the practical application of light field imaging.

[0067] To address the aforementioned problems, this embodiment discloses a spatial super-resolution method for light field images based on a multi-scale large kernel attention mechanism, comprising:

[0068] S1: Obtain the target light field image to be reconstructed;

[0069] S2: Input the target light field image into the trained spatial super-resolution reconstruction model, and output the spatial super-resolution reconstructed image of the target light field image;

[0070] Combination Figure 1 As shown, the processing steps during the training of the spatial super-resolution reconstruction model include:

[0071] S201: Convert high-resolution light field images into low-resolution light field images and use them as training samples to input into the spatial super-resolution reconstruction model;

[0072] S202: The input light field image is processed by a shallow feature extraction network (shallow feature extraction) to generate a shallow feature map;

[0073] S203: Using a large kernel feature extraction network (deep feature extraction), angle features, spatial features, and epipolar plane image features are extracted from the macro-pixel image of the shallow feature map, the sub-aperture image (SAI) that generates the macro-pixel feature map (i.e., the angle low-frequency feature map in subsequent step S2031), and the epipolar plane image of the shallow feature map, and then fused to generate a 4D light field feature map.

[0074] S204: Upsample and reconstruct the 4D light field feature map through an upsampling reconstruction network (feature upsampling) to generate a spatial super-resolution reconstructed image of the input light field image;

[0075] S205: Calculate the training loss based on the spatial super-resolution reconstructed image of the input light field image and the corresponding high-resolution light field image, and optimize the parameters of the spatial super-resolution reconstruction model based on the training loss;

[0076] S206: Repeat steps S201 to S205 to iteratively train the spatial super-resolution reconstruction model until the model converges or reaches the preset number of iterations.

[0077] S3: The spatial super-resolution reconstructed image output by the spatial super-resolution reconstruction model is used as the super-resolution reconstruction result of the target light field image. Specifically, super-resolution is performed on the Y channel of the light field image, while the Cb and Cr channels are reconstructed using bicubic interpolation.

[0078] It should be noted that image super-resolution (SR) refers to the technique of reconstructing a high-resolution (HR) image from an existing low-resolution (LR) image using image processing and machine learning techniques. Specifically, a light field image can be represented as three 2D subspace image forms: sub-aperture image (SAI), macro-pixel image (MacPI), and epipolar-plane image (EPI), and these three 2D subspace image forms can be converted into each other. This invention represents the light field image as a sub-aperture image array as network input. The light field can be represented as... Where C is the number of channels, B is the batch size, U and V are the angular resolutions, and H and W are the spatial resolutions. The goal of light field image super-resolution reconstruction is to reconstruct light field images from low-resolution (LR) light field images. Reconstructing super-resolution (SR) light field images , This represents the spatial resolution magnification factor.

[0079] This invention designs a hybrid CNN-Transformer architecture. It extracts low-frequency angular information through an angle Transformer and captures low-frequency spatial information through multi-scale large-kernel convolution. Secondly, based on the non-local correlation of spatial angles, this invention proposes a large-kernel attention unit (LKAM) that expands the receptive field of feature extraction, applied to epipolar images. Furthermore, in this invention's method, dense connection operations are introduced into the large-kernel feature extraction module and encapsulated into a dense large-kernel attention module (DLKAM), applied to the spatial feature extraction part of the light field image. Finally, this invention combines DLKAM, LKAM, and the angle Transformer to form a large-kernel feature extraction module, used to achieve efficient processing of three 2D subspace images (SAI, MacPI, and EPI) of the light field image transformation. By cascading multiple large-kernel feature extraction modules to form a multi-dimensional feature extraction network, it deeply explores the spatial, angular, and epipolar plane feature information of the light field image, obtaining deep-level intrinsic features of the light field, thereby achieving high-quality reconstruction of super-resolution light field images.

[0080] To better illustrate the technical solution of the present invention, this embodiment will be described in more detail through the following parts.

[0081] I. Image Preprocessing

[0082] In the specific implementation process, the light field image data used for training and testing the spatial super-resolution reconstruction model can be selected from public light field image databases, such as EPFL, HCI_new, HCI_old, INRIA_Lytro, and Stanford_Gantry. After obtaining high-resolution light field image data from public light field image databases, the high-resolution light field images are cropped, and bicubic interpolation is used to downsample the cropped light field images to obtain low-resolution light field images as training samples. In this embodiment, 144 scenes were selected from the above-mentioned light field image database for training, and 23 scenes were selected for testing. The original angular resolution of all selected high-resolution light field images was 9×9. During training and testing, the angular resolution of the original light field images was cropped to 5×5, and the spatial resolution of the original light field images was cropped to 64×64 image blocks with a stride of 32. Then, bicubic interpolation was used to downsample the spatial resolution of the image blocks to 32×32. Thus, the low-resolution light field images obtained by processing and converting the high-resolution light field images used for training and testing scenes were used as training samples and testing samples, respectively.

[0083] II. Shallow Feature Extraction Network

[0084] In the specific implementation process, in order to facilitate subsequent light field image feature extraction, this invention designs an initialization shallow feature extraction network, which takes the original low-resolution light field image input into the spatial super-resolution reconstruction model. Mapped to shallow features ,in, It is the number of channels, Is it batch quantity? and It is angular resolution. and It refers to spatial resolution.

[0085] Specifically, the shallow feature extraction network consists of cascaded 3×3 convolutional layers and LeakyReLU activation function layers.

[0086] III. Large Kernel Feature Extraction Network (LKFEM)

[0087] CNNs excel at local feature extraction and are computationally efficient, but they are weak in modeling global dependencies. In contrast, Transformers can effectively capture long-range dependencies through self-attention mechanisms, but they suffer from higher computational costs and insufficient capture of local details. Therefore, existing light field spatial super-resolution methods based on CNNs and Transformers cannot simultaneously extract global and local information from light field images, resulting in degraded image reconstruction performance.

[0088] To address this problem, this invention designs a hybrid CNN-Transformer structure, namely the Large Kernel Feature Extraction Network (LKFEM). For example... Figure 1 As shown in (b), in the top branch of LKFEM, the present invention first sets the initial features Remodeling into intermediate features Secondly, the reshaped features Input into two cascaded angle Transformers.

[0089] This process can be described as follows:

[0090] ;

[0091] In the formula: and Both are angle Transformers. This represents the input to the cascaded angle Transformer. This represents the output of the cascaded angle Transformer.

[0092] To extract rich spatial high-frequency information from sub-aperture images, this invention will Remodeling and will Input a dense big-kernel attention module (DLKAM) to generate spatial features. ; then on Perform 1×1 two-dimensional convolution To achieve dimensionality reduction and generate features The process can be described as follows:

[0093] ;

[0094] ;

[0095] in, This represents the input features of the dense large-kernel attention module. DLKAM(.) represents the output features of the dense large kernel attention module. This represents the output of the dimensionality-reduced convolution.

[0096] In the bottom branch of LKFEM, Reconstructed into horizontal epipolar image features Considering the nonlocality of pixel motion, a single cascaded convolution cannot effectively extract the rich sub-pixel information in epipolar images. Therefore, this invention employs two cascaded LKAMs to expand the receptive field when extracting features from horizontal epipolar images. This design ensures that the model fully models the spatial-angular features of the light field and maintains its geometric consistency. Subsequently, this invention processes the extracted horizontal epipolar image features... Reconstructed as vertical epipolar image features The vertical epipolar image features were obtained by processing them using the same two cascaded LKAM algorithms. .

[0097] This process can be described as follows:

[0098] ;

[0099] ;

[0100] Where LKAM(.) represents a large-kernel attention unit, This represents the input to a cascaded (horizontal) large-kernel attention unit. This represents the output of a cascaded (horizontal) large-kernel attention unit. This represents the input to a cascaded (vertical) large-kernel attention unit. This represents the output of a cascaded (vertical) large kernel attention unit.

[0101] like Figure 1 As shown in (b), the processing steps of the large kernel feature extraction network include:

[0102] S2031: Reshape the input shallow feature map into a macro-pixel image, and input it into two cascaded angle Transformers for angle feature extraction to generate a low-frequency angle feature map;

[0103] S2032: Reshape the low-frequency feature map of the angle into a sub-aperture image and input it into the dense large kernel attention module for spatial feature extraction of the high-frequency feature map of the space;

[0104] S2033: After inputting the high-frequency spatial feature map into a 1×1 convolutional layer for dimensionality reduction, it is reshaped into a spatial-angular feature map with the same tensor as the input shallow feature map.

[0105] S2034: Reshape the input shallow feature map into a horizontal epipolar image, and input it into two cascaded large-kernel attention units for epipolar plane image feature extraction to generate a horizontal epipolar feature map;

[0106] S2035: Reshape the horizontal epipolar feature map into a vertical epipolar image, and input it into two cascaded large-kernel attention units for epipolar plane image feature extraction to generate a vertical epipolar feature map;

[0107] S2036: Reshape the vertical epipolar feature map into an epipolar planar image feature map with the same tensor as the input shallow feature map;

[0108] S2037: The spatial-angle feature map and the epipolar plane image feature map are concatenated and fed into a 1×1 convolutional layer for dimensionality reduction. Then, they are added element-wise to the input shallow feature map to generate the 4D light field feature map output by the large kernel feature extraction network.

[0109] 1. Angle Transformer

[0110] The Angle Transformer consists of a cascaded first normalization layer, a multi-head self-attention layer, a second normalization layer, and a multilayer perceptron.

[0111] The input to the Angle Transformer is element-wise summed with the output of the multi-head self-attention mechanism to generate the input to the second normalized layer. The input to the second normalized layer is then element-wise summed with the output of the multilayer perceptron to generate the output features of the Angle Transformer. Let the input feature map of the Angle Transformer be... The intermediate feature map is The output feature map is The angle Transformer calculation process can be described as follows:

[0112] ;

[0113] ;

[0114] in, This indicates the bulls' self-attention. This represents a multilayer perceptron.

[0115] 2. Dense Large Kernel Attention Module (DLKAM)

[0116] Existing learning-based optical field spatial super-resolution methods lose spatial information as network depth increases, implicitly limiting the model's performance and potential. This invention introduces DLKAM as a spatial feature extraction component. For example... Figure 1 As shown in (c), this module aggregates the spatial features extracted from each layer through dense connections in SAI. Therefore, shallow detail features and deep semantic features can interact fully, avoiding the loss of inter-layer spatial information caused by network deepening. Specifically, DLKAM uses LKAM units (LKAM-1 to LKAM-4) as basic modules, constructing hierarchical feature transmission paths through a dense connection architecture. Assume the input of the i-th LKAM unit is X. i The output is Y iThen the feature interactions between units satisfy:

[0117] ;

[0118] in, This indicates a feature concatenation operation. This is the i-th large kernel attention module unit. This densely connected structure enables direct transfer of spatial information between layers by allowing each LKAM unit to input the output features of all shallow units. This design effectively preserves more complete spatial details in the sub-aperture image for subsequent feature utilization, thereby improving the reconstruction performance of the light field image.

[0119] like Figure 1 As shown in (c), the dense large-kernel attention module includes several cascaded large-kernel attention units;

[0120] For each large kernel attention unit, its output is concatenated with the outputs of all preceding large kernel attention units and the input of the dense large kernel attention module to generate the input of the next large kernel attention unit; the output of the last large kernel attention unit is concatenated with the outputs of all preceding large kernel attention units and the input of the dense large kernel attention module to generate the output feature map of the dense large kernel attention module.

[0121] 3. Large Kernel Attention Unit (LKAM)

[0122] To expand the receptive field of light field image spatial features extracted by convolution, this invention proposes a light field spatial super-resolution network with LKAM as the core feature extraction unit. Figure 1 (d) illustrates the structure of LKAM. This module combines a classic multi-scale mechanism with an emerging large-kernel convolution algorithm. The multi-scale mechanism can implicitly encode features from coarse to fine, enabling it to capture the spatial-angular characteristics of light field images across different receptive fields. Simultaneously, the large-kernel convolution establishes correlations between different ranges of the light field image while maintaining the module's lightweight design. The computation process of LKAM is similar to that of a Transformer; let its input features be... Intermediate features are Output features are The calculation process of LKAM can be described as follows:

[0123] ;

[0124] ;

[0125] in, The representation layer is normalized, MKDAM represents the multi-branch kernel decomposition attention module, and DCGM represents the bi-branch convolutional gating module.

[0126] like Figure 1 As shown in (d), the large kernel attention unit includes a cascaded first normalization layer, a multi-branch kernel decomposition attention module, a second normalization layer, and a dual-branch convolutional gating module;

[0127] The input of the large kernel attention unit is added element-wise to the output of the multi-branch kernel decomposition attention module to generate the input of the second normalized layer; the input of the second normalized layer is added element-wise to the output of the bi-branch convolutional gating module to generate the output feature of the large kernel attention unit.

[0128] 4. Multi-branch kernel decomposition attention module (MKDAM)

[0129] Transformer-based spatial super-resolution methods for light fields often use multi-head self-attention as the self-attention component. However, conventional MHSA struggles to capture local spatial details and overemphasizes global features, leading to high computational costs. The coupling of spatial and angular information in sub-aperture images inherently increases the difficulty of comprehensive spatial feature extraction. Furthermore, due to the nonlocality of pixel motion in epipolar images, a larger receptive field is needed to extract the spatial-angular features of the light field. To address these issues, this invention designs a multi-branch kernel decomposition attention module. This module can capture spatial pixel features over a wider range of images and adapts to the spatial-angular characteristics of the light field.

[0130] like Figure 1 As shown in (d), taking the MKDAM in the first LKAM of the horizontal epipolar image (EPI-H) as an example, let the features of the input MKDAM be... First, initial features are extracted through pointwise convolution and then increased in dimensionality to generate features. ; then on Perform channel segmentation to obtain features and The process can be described as follows:

[0131] ;

[0132] ;

[0133] in, This represents pointwise convolution. This indicates a channel splitting operation.

[0134] Features The input to MKDAM is first divided into four equal parts along the channel dimension. MKDAM contains four parallel branches, which are obtained by large kernel convolution decomposition with kernel sizes of 11×11, 23×23, 39×39, and 89×89, respectively. Let the input feature of the i-th branch be... , Convolution can be separated by the decomposed depth. Depth-separable dilated convolution (Where the dilation rate d is taken as 2, 3, 4, and 5 respectively) and pointwise convolution Generate attention map Finally, the attention map generated for each branch is weighted... Weighting ( Depend on Depth-separable convolution get).

[0135] This process can be represented as:

[0136] ;

[0137] ;

[0138] ;

[0139] in, This indicates a channel splicing operation. This represents the output features after channel concatenation. While large-kernel convolutional attention can capture pixel features within a larger receptive field, it may produce block artifacts. To address this issue and enhance the local continuity of the light field image, this invention employs spatially gated attention. Let the output features of spatially gated attention be... Then the process can be represented as:

[0140] .

[0141] like Figure 1 As shown in (d), the processing steps of the multi-branch kernel decomposition attention module include:

[0142] 1) Input the input feature map into a 1×1 pointwise convolutional layer for feature extraction and dimensionality upscaling to generate an initial feature map;

[0143] 2) Perform channel segmentation on the initial feature map to obtain the first segmentation feature map and the second segmentation feature map;

[0144] 3) Input the first segmentation feature map into the four parallel branches for attention enhancement, and then concatenate the outputs of the four branches to generate a fused feature map;

[0145] Each branch includes cascaded depthwise convolutional layers and depthwise dilated convolutional layers (the dilation rate of the four branches is...). The convolutional layers are 2, 3, 4, and 5 respectively, and the output feature map of each branch is generated by multiplying the output of the depthwise convolutional layer and the output of the pointwise convolutional layer (i.e., the weights of the attention map) element by element.

[0146] 4) After the second segmentation feature map is input into the pointwise convolutional layer for processing, it is multiplied elementwise with the fused feature map to generate the output feature map of the multi-branch kernel decomposition attention module.

[0147] 5. Dual-branch convolutional gating module (DCGM)

[0148] Existing Transformer-based light field spatial super-resolution methods often use computationally expensive Multilayer Perceptrons (MLPs) as feedforward networks. While MLPs can effectively enhance local feature representations, they prolong model training time, thus affecting their generalization ability. To address this issue, this invention designs a dual-branch convolutional gating module (DCGM) consisting of pointwise convolutions and depthwise separable convolutions, such as... Figure 1 As shown in (e).

[0149] like Figure 1 As shown in (e), the dual-branch convolutional gating module includes a feature branch and a gating branch;

[0150] Feature branches include pointwise convolutional layers;

[0151] Gated branches include cascaded pointwise convolutional layers and depthwise convolutional layers;

[0152] After the outputs of the first branch and the second branch are multiplied element-wise, they are input into the pointwise convolutional layer for processing to generate the output feature map of the dual-branch convolutional gating module.

[0153] Specifically, the processing steps of the dual-branch convolutional gating module include:

[0154] 1) Input the input feature map into the feature branch, and extract the initial features through a pointwise convolutional layer to obtain the feature map. ;

[0155] 2) Input the feature map into the gated branch, and extract the initial features through pointwise convolutional layers to obtain the feature map. Subsequently, spatial context information is captured through depthwise convolutional layers to generate a spatial gating weight map. ;

[0156] The formula is expressed as:

[0157] ;

[0158] In the formula, DWConv 7×7 This represents a 7×7 depthwise convolution.

[0159] 3) Feature map Spatial Gating Weight Map Perform element-wise multiplication to generate gated modulation. ;

[0160] The formula is expressed as:

[0161] ;

[0162] In the formula: This represents element-wise multiplication;

[0163] 4) Gating modulation After cross-channel weighted aggregation of the input pointwise convolutional layer, a residual connection is formed with the input feature map, and then processed through learnable parameters. Adaptively adjust attention weight strength to generate output feature maps of the bi-branch convolutional gating module. ;

[0164] The formula is expressed as:

[0165] ;

[0166] In the formula: This indicates a pointwise convolutional layer. This represents the input to the bi-branch convolutional gating module.

[0167] IV. Upsampling Reconstruction Network

[0168] The upsampling reconstruction network (feature upsampling) upsamples the 4D light field features of the light field image and then adds them to the feature map obtained by bicubic interpolation of the light field image input to the model to generate a spatial super-resolution reconstructed image of the light field image as the model output.

[0169] The upsampling reconstruction network consists of a cascaded first 1×1 convolutional layer, a pixel rearrangement layer, and a second 1×1 convolutional layer;

[0170] The 4D light field feature map input from the upsampling module is used as the input to the first 1×1 convolutional layer, increasing the number of channels in the 4D light field feature map to [number missing]. The output of the first 1×1 convolutional layer is used as the input to the pixel rearrangement layer to reshape the output features of the first 1×1 convolutional layer from the channel dimension to the spatial dimension, thereby generating a super-resolution feature map. ; super-resolution feature map The input to the second 1×1 convolutional layer is used for super-resolution feature refinement, generating the spatial super-resolution reconstructed image output by the upsampling module; where... This represents the spatial resolution magnification factor.

[0171] V. Model Training

[0172] In the specific implementation process, the spatial super-resolution reconstruction model is trained in the following way: the partitioned training set is used as the input of the spatial super-resolution reconstruction model, a loss function that is robust to discrete values ​​is constructed, and the model image of the spatial super-resolution reconstruction model is reconstructed with the goal of minimizing the loss function, thereby training the spatial super-resolution reconstruction model.

[0173] Specifically, the training loss is calculated using the following loss function:

[0174] ;

[0175] In the formula: Represents the loss function; Indicates the first Low-resolution light field images of the training samples Spatial super-resolution reconstructed images; Represents the training sample set, Indicates the first Low-resolution light field images of the training samples The corresponding high-resolution light field image, This indicates the number of training samples contained in the training sample set.

[0176] VI. Experimental Instructions

[0177] To better illustrate the advantages of the technical solution of the present invention, the following experiment is disclosed in this embodiment.

[0178] This experiment uses two light field synthesis databases, HCInew and HCIold, and three real-world scene databases, EPFL, INRIA, and STFgantry, for training and testing. All datasets have an angular resolution of 9×9. This experiment extracts only the central 5×5 light field images for training and testing. During training, bicubic interpolation is used for 0.5 / 0.25 downsampling to generate 2× / 4×LR training samples of size 32×32 (H=W=32). Furthermore, data augmentation techniques such as horizontal and vertical flipping and 90-degree rotation are applied to enhance the generalization ability of the data. During testing, the light field images are cropped into image patches, and SR (super-resolution reconstruction) is performed individually on these patches. Subsequently, the SR-reconstructed image patches are aggregated to generate a high-resolution light field image. To quantitatively evaluate reconstruction accuracy, this experiment uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) as evaluation metrics.

[0179] In this experimental example, the number of DLKAMs in the LKAMs was set to 4, and the number of LKFEMs was set to 5. In all experiments, the Adam optimizer was used for training, with β1=0.99 and β2=0.999. The convolutional kernel weights were initialized using the Xavier algorithm. During training, the batch size was 1, and the initial learning rate was 2×10⁻⁶. −4 The learning rate was halved every 10 epochs, and training lasted for a total of 40 epochs. The network in this experiment was implemented in the PyTorch framework and trained on a PC equipped with a single NVIDIA RTX5080 GPU.

[0180] To evaluate the performance of the method of this invention, it was compared with some state-of-the-art methods on five publicly available datasets. Fourteen light field spatial SR methods were included. All methods were tested under the same experimental conditions to ensure a fair comparison.

[0181] Table 1. Comparison of PSNR (dB) / SSIM for different methods in 2× / 4× tasks.

[0182]

[0183] Table 1 lists the results of different methods on five widely used datasets at different scaling factors. On 2×SR, the method of this invention achieves the best results in terms of average PSNR / SSIM, and also achieves the best performance on the EPFL and INRIA datasets. Specifically, compared to DistgSSR, the method of this invention improves average PSNR / SSIM by 0.72 dB and 0.0018, respectively; compared to LFT, the method of this invention achieves performance gains of 0.81 dB and 0.0022 in average PSNR / SSIM; and compared to L2FMamba, the method of this invention achieves performance gains of 0.17 dB and 0.0012 in average PSNR / SSIM. Compared to CSFNet, the method of this invention improves average PSNR / SSIM by 0.07 dB and 0.0006, respectively. Although the method of this invention performs poorly on the STFgantry dataset, it achieves the best average PSNR / SSIM, and also demonstrates the best or near-best PSNR / SSIM on the other four datasets. This fully demonstrates that the method of this invention achieves a good trade-off between global and local information extraction capabilities. These results clearly indicate that the method of this invention is competitive in 2×SR tasks.

[0184] The method presented in this invention demonstrates suboptimal or optimal results on complex 4×SR tasks. Specifically, compared to DistgSSR, the method improves the average PSNR / SSIM by 0.66 dB and 0.0046, respectively; compared to LFT, the method achieves the best PSNR / SSIM on the three large disparity datasets HCInew, HCIold, and STFgantry, fully demonstrating its superior global information modeling capability; compared to L2FMamba, the method improves the average PSNR / SSIM by 0.15 dB and 0.0018, respectively, demonstrating the competitiveness of the CNN+Transformer architecture over the Mamba architecture; compared to CSFNet, the method decreases the average PSNR by 0.08 dB on the HCInew dataset, but improves the average PSNR / SSIM by 0.11 dB and 0.0012, respectively. These results clearly demonstrate the significant competitiveness and advantages of the method presented in 4×SR tasks with high image degradation.

[0185] This experiment conducted ablation experiments on the proposed modules. As shown in Table 2, Model "1" represents replacing DCGM with MLP, Model "2" represents replacing MKDAM with multi-head self-attention, and Model "3" represents the method of this invention. The table shows that Model "3" exhibits the best performance gain in average PSNR / SSIM. Specifically, compared to Model "1", Model "3" improves average PSNR / SSIM by 0.21dB and 0.0019 respectively, with lower parameter count and FLOPs. This indicates that DCGM can effectively reduce model complexity while enhancing feature representation. Compared to Model "2", although Model "3" has increased parameter count and FLOPs, it achieves a performance gain of 0.52dB and 0.0041 in average PSNR / SSIM. This shows that MKDAM effectively models local image details using large kernel convolution while extracting global information from the light field image through multi-scale properties, thereby improving the network's ability to express local details of the light field image.

[0186] Table 2 Ablation studies of different components in 4×SR

[0187]

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A light field image spatial super-resolution method based on a multi-scale large kernel attention mechanism, characterized in that, The method comprises the following steps: S1: obtaining a target light field image to be reconstructed; S2: inputting the target light field image into a trained spatial super-resolution reconstruction model to output a spatial super-resolution reconstruction image of the target light field image; The processing steps for training the spatial super-resolution reconstruction model comprise: S201: converting a high-resolution light field image into a low-resolution light field image as a training sample input to the spatial super-resolution reconstruction model; S202: performing feature extraction on the input light field image through a shallow feature extraction network to generate a shallow feature map; S203: extracting angle features, spatial features and epipolar plane image features from a macro-pixel image of the shallow feature map, a sub-aperture image for generating a macro-pixel feature map and an epipolar plane image of the shallow feature map through a large-core feature extraction network, and fusing to generate a 4D light field feature map; S204: performing up-sampling reconstruction on the 4D light field feature map through an up-sampling reconstruction network to generate a spatial super-resolution reconstruction image of the input light field image; S205: calculating a training loss based on the spatial super-resolution reconstruction image of the input light field image and the corresponding high-resolution light field image, and optimizing the parameters of the spatial super-resolution reconstruction model based on the training loss; S206: repeating steps S201 to S205 to iteratively train the spatial super-resolution reconstruction model until the model converges or a preset number of iterations is reached; S3: taking the spatial super-resolution reconstruction image output by the spatial super-resolution reconstruction model as a super-resolution reconstruction result of the target light field image.

2. The light field image spatial super-resolution method based on multi-scale large kernel attention mechanism according to claim 1, characterized in that: In step S203, the processing steps of the large-core feature extraction network comprise: S2031: reshaping the input shallow feature map into a macro-pixel image and inputting it into two cascaded angle Transformers for angle feature extraction to generate an angle low-frequency feature map; S2032: reshaping the angle low-frequency feature map into a sub-aperture image and inputting it into a dense large-core attention module for spatial feature extraction to generate a spatial high-frequency feature map; S2033: inputting the spatial high-frequency feature map into a 1x1 convolution layer for dimension reduction, and reshaping it into a space-angle feature map with the same tensor as the input shallow feature map; S2034: reshaping the input shallow feature map into a horizontal epipolar image and inputting it into two cascaded large-core attention units for epipolar plane image feature extraction to generate a horizontal epipolar feature map; S2035: reshaping the horizontal epipolar feature map into a vertical epipolar image and inputting it into two cascaded large-core attention units for epipolar plane image feature extraction to generate a vertical epipolar feature map; S2036: reshaping the vertical epipolar feature map into an epipolar plane image feature map with the same tensor as the input shallow feature map; S2037: concatenating the space-angle feature map and the epipolar plane image feature map, inputting them into a 1x1 convolution layer for dimension reduction, and then element-wise adding them with the input shallow feature map to generate a 4D light field feature map output by the large-core feature extraction network.

3. The method of claim 2, wherein the method is based on a multi-scale large kernel attention mechanism. In step S2031, the angle Transformer comprises a first normalization layer, a multi-head self-attention, a second normalization layer and a multi-layer perceptron which are cascaded. The input of the angle Transformer is element-wise added with the output of the multi-head self-attention to generate the input of a second normalization layer; the input of the second normalization layer is element-wise added with the output of the multi-layer perception to generate the output feature of the angle Transformer.

4. The light field image spatial super-resolution method based on the multi-scale large kernel attention mechanism according to claim 2, characterized in that: In step S2032, the dense large kernel attention module includes a plurality of large kernel attention units in cascade; For each large kernel attention unit, the output thereof is spliced with the outputs of all the preceding large kernel attention units and the input of the dense large kernel attention module to generate the input of the next large kernel attention unit; the output of the last large kernel attention unit is spliced with the outputs of all the preceding large kernel attention units and the input of the dense large kernel attention module to generate the output feature map of the dense large kernel attention module.

5. The method of claim 4, wherein the method is based on a multi-scale large kernel attention mechanism. The large kernel attention unit includes a first normalization layer, a multi-branch kernel decomposition attention module, a second normalization layer and a double-branch convolution gating module in cascade; The input of the large kernel attention unit is element-wise added with the output of the multi-branch kernel decomposition attention module to generate the input of the second normalization layer; the input of the second normalization layer is element-wise added with the output of the double-branch convolution gating module to generate the output feature of the large kernel attention unit.

6. The method of claim 5, wherein the method is based on a multi-scale large kernel attention mechanism. The processing steps of the multi-branch kernel decomposition attention module include: 1) inputting the input feature map into a 1x1 point-wise convolution layer for feature extraction and dimensionality increase to generate an initial feature map; 2) performing channel segmentation on the initial feature map to obtain a first segmented feature map and a second segmented feature map; 3) inputting the first segmented feature map into four parallel branches respectively for attention enhancement, and splicing the outputs of the four branches to generate a fusion feature map; Each branch includes a depth-wise convolution layer, a depth-wise dilated convolution layer and a point-wise convolution layer in cascade, and the output of the depth-wise convolution layer and the output of the point-wise convolution layer in each branch are element-wise multiplied to generate the output feature map of the branch; 4) inputting the second segmented feature map into a point-wise convolution layer for processing, and then element-wise multiplying the fusion feature map to generate the output feature map of the multi-branch kernel decomposition attention module.

7. The light field image spatial super-resolution method based on multi-scale large kernel attention mechanism according to claim 5, characterized in that: The double-branch convolution gating module includes a feature branch and a gating branch; The feature branch includes a point-wise convolution layer; The gating branch includes a point-wise convolution layer and a depth-wise convolution layer in cascade; The output of the first branch and the output of the second branch are element-wise multiplied, and then input into a point-wise convolution layer for processing to generate the output feature map of the double-branch convolution gating module.

8. The method of claim 7, wherein the method is based on a multi-scale large kernel attention mechanism. The processing steps of the double-branch convolution gating module include: 1) input the input feature map into the feature branch, extract the initial feature through the point-by-point convolution layer, and obtain the feature map ; 2) input the input feature map into the gating branch, and obtain a feature map by extracting initial features through a point-by-point convolution layer After that, spatial context information is captured through a depth-by-depth convolution layer to generate a spatial gating weight map ; The formula is represented as: ； In the formula, DWConv 7×7 represents 7 x 7 depth-wise convolution 3) multiply the feature map with the spatial gating weight map element-wise to generate the gated modulation ; The formula is represented as: ； In the formulae: denotes an element-wise multiplication; 4) gating modulation After the cross-channel weighted aggregation of the input point-wise convolution layer, the residual connection is performed with the input feature map, and the learnable parameters Adaptive adjustment of attention weight strength to generate output feature map of double-branch convolution gating module ; The formula is represented as: ； In the formula: represents a point-by-point convolution layer, represents the input of the double-branch convolution gate module.

Images