Self-attention and state space mixed light field image super-resolution reconstruction method

Abstract

The invention discloses a self-attention and state space mixed light field image super-resolution reconstruction method, which relates to the technical field of image super-resolution and comprises the following steps of: extracting shallow layer features by adopting convolution operation; constructing a first basic feature extraction unit in combination with a multi-branch structure and a state space model, and constructing a second basic feature extraction unit in combination with a multi-sub attention mechanism and a channel multi-layer perception mechanism; extracting spatial features and angle features based on the first basic features; extracting a polar plane feature based on the second basic feature; calculating deep features by combining the spatial features, the angle features and the polar plane features; and calculating a high-resolution light field image based on the shallow layer features, the deep layer features and the input low-resolution light field image. According to the method, self-attention and state space are introduced to model the coupling space, angle and parallax relation of complex light field data, the characterization capability of global space-angle-parallax information in the light field image super-resolution process is remarkably enhanced, and the reconstruction quality of high-frequency details is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image super-resolution technology, and in particular to a method for super-resolution reconstruction of light field images that combines self-attention and state space. Background Technology

[0002] Due to the inherent spatial resolution trade-offs in light field imaging, cameras must reduce the spatial resolution of each sub-aperture image (SAI) when acquiring multi-view information, which greatly restricts the further expansion of light field images in high-precision vision applications. Therefore, Light Field Image Super-Resolution (LFSR) technology has become a key solution. Its core objective is to improve the spatial resolution of low-resolution light field images, fully exploit the complementary spatial information within the light field data, and provide high-quality data support for subsequent visual analysis and advanced processing tasks.

[0003] Deeply exploring the information within complex four-dimensional light fields (LF) is crucial for advancing light field super-resolution (LFSR) technology. Compared to conventional image super-resolution techniques, the challenge of light field super-resolution lies in fully utilizing the inherent spatial and angular information within light field images to improve reconstruction accuracy. In recent years, both Convolutional Neural Network (CNN)-based and Transformer-based techniques have achieved significant results in the LFSR field. These techniques typically decompose the four-dimensional light field into a two-dimensional subspace for processing, limiting feature learning to specific aspects such as disparity cues or spatial angular correlations. This significantly improves the spatial resolution of light field images compared to traditional methods. However, CNN-based LFSR methods can only handle local spatial angular features of light field images; while Transformer-based LFSR methods can explore non-local spatial angular features, Transformer alone cannot efficiently and comprehensively explore the intrinsic structural characteristics of light field images.

[0004] State-space models, as an emerging sequence modeling architecture, can achieve a true global receptive field with linear computational complexity, while effectively preserving local information and improving feature representation efficiency. This perfectly solves the dilemma of weak global dependencies in CNNs and the computational explosion and fragmented local information in Transformers. Therefore, this study attempts a hybrid architecture of Transformer and state-space models to efficiently capture the inherent long-range dependencies in the 4D structure of light field images, while simultaneously achieving deep interaction between spatial and angular information. Chinese invention application CN119809940A discloses a light field image super-resolution reconstruction method based on a state-space model. By utilizing the dynamic characteristics of the state-space model, it significantly improves the global spatial angle information representation and detail reconstruction capabilities of the light field image super-resolution method. However, it has the following shortcomings: First, this method relies on a single state-space model to uniformly model the spatial, angular, and polar plane features of the light field image, lacking the ability to specifically model polar plane disparity features. As a sequential modeling architecture, the state-space model is difficult to fully capture the global disparity correlation in the polar plane image. Second, the deep feature extraction unit of this method directly feeds the input features into the selective state-space model for processing without performing multi-branch fine decomposition of the features, resulting in insufficiently fine extraction granularity of global and local features, and the effective difference information between features is not fully explored. Summary of the Invention

[0005] The purpose of this invention is to solve the problems in the prior art.

[0006] The technical solution adopted by this invention to solve its technical problem is: to provide a method for super-resolution reconstruction of light field images by hybrid self-attention and state space, comprising the following steps:

[0007] A shallow feature extraction module is constructed using convolution operations to extract shallow features from the input low-resolution light field image;

[0008] The first basic feature extraction unit is constructed by combining a multi-branch structure and a state-space model, and the second basic feature extraction unit is constructed by combining a multi-head sub-attention mechanism and a channel multilayer perceptron.

[0009] Based on the first basic feature extraction unit, a spatial feature Mamba extraction submodule and an angular feature Mamba extraction submodule are constructed to extract spatial features and angular features from the shallow features, respectively; based on the second basic feature extraction unit, a polar plane feature Transformer extraction submodule is constructed to extract polar plane features from the shallow features.

[0010] A deep feature extraction module is constructed based on the spatial feature Mamba extraction submodule, the angular feature Mamba extraction submodule, and the polar plane feature Transformer extraction submodule; the deep feature extraction module extracts deep features based on the shallow features;

[0011] An image reconstruction module is constructed to obtain a high-resolution light field image based on the shallow features, the deep features, and the input low-resolution light field image.

[0012] By combining a shallow feature extraction module, a deep feature extraction module, and an image reconstruction module, a super-resolution network is constructed to achieve super-resolution reconstruction of light field images.

[0013] Preferably, the operation of the first basic feature extraction unit is represented as follows: This includes the following operations:

[0014] ;

[0015] ;

[0016] in, , and These represent the input features, intermediate features, and output features of the first basic feature extraction unit, respectively. The function of the presentation layer is normalized. express Convolutional layer; This represents a depthwise convolutional feedforward layer, consisting of sequentially connected... Convolution, depthwise separable convolution and Convolution; , This represents a learnable residual connection. This represents the state space mechanism for perceiving empty angle features.

[0017] Preferably, the empty angle feature perception state space mechanism The expression is as follows:

[0018] ;

[0019] in, Represents a linear mapping layer. This represents depthwise separable convolution. This represents the activation function. This represents a multi-branch state-space mechanism. This represents the Hadama dot product;

[0020] Multi-branch state space mechanism The calculation process includes the following steps:

[0021] Channel division is represented as:

[0022] ;

[0023] The local feature extraction branch is represented as follows:

[0024] ;

[0025] The global feature extraction branch is represented as:

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] Feature fusion, represented as:

[0031] ;

[0032] in, This represents the input to the multi-branch state-space mechanism. This indicates a channel partitioning operation. This represents the intermediate features after channel division; Local features extracted by the multi-branch state-space mechanism; This represents the global average pooling operation. This represents the output feature after global average pooling operation; Indicates an upsampling operation. Representing a state-space model, This represents the output feature after upsampling and state-space model processing; This represents the features extracted after differentiation. Global features extracted using a multi-branch state-space mechanism; This indicates that features will be fused by channel. This represents the output feature after fusing global and local features.

[0033] Preferably, the operation of the spatial feature Mamba extraction submodule is represented as follows: The calculation process is as follows:

[0034] ;

[0035] ;

[0036] in, , and These represent the input features, intermediate features, and output features of the spatial feature Mamba extraction submodule, respectively. This represents a matrix dimensionality transformation operation that converts input features into sub-aperture image features. express Convolutional layer.

[0037] Preferably, the operation of the angle feature Mamba extraction submodule is represented as follows: The calculation process is as follows:

[0038] ;

[0039] ;

[0040] in, , and These represent the input features, intermediate features, and output features of the angle feature Mamba extraction submodule, respectively. This represents a matrix dimension transformation operation that converts sub-aperture image features into macro-pixel image features. This represents a matrix dimension transformation operation that converts the macro-pixel image features into sub-aperture image features; express Convolutional layer.

[0041] Preferably, the operation of the second basic feature extraction unit is represented as follows: This includes the following operations:

[0042] ;

[0043] ;

[0044] in, , and These represent the input features, intermediate features, and output features of the second basic feature extraction unit, respectively. express Convolutional layer This represents the activation function. This represents the self-attention mechanism for parallax perception.

[0045] Preferably, the parallax perception self-attention mechanism The calculation process is as follows:

[0046] ;

[0047] ;

[0048] in, , and These represent the input features, intermediate features, and output features of the parallax perception self-attention mechanism, respectively. Represents a multi-head self-attention mechanism. The function of the presentation layer is normalized. This refers to a multilayer sensor for the channel.

[0049] Preferably, the polar plane feature Transformer extraction submodule is represented as follows: The calculation process is as follows:

[0050] ;

[0051] ;

[0052] in, , and These represent the input features, intermediate features, and output features of the polar plane feature extraction submodule of the Transformer. This represents a matrix dimension transformation operation that converts sub-aperture image features into horizontal epipolar plane image features. This represents a matrix dimension transformation operation that converts horizontal epipolar plane image features into sub-aperture image features; This represents a matrix dimension transformation operation that converts sub-aperture image features into vertical epipolar plane image features. This represents a matrix dimension transformation operation that converts vertical epipolar plane image features into sub-aperture image features.

[0053] Preferably, the construction of the deep feature extraction module based on the spatial feature Mamba extraction submodule, the angular feature Mamba extraction submodule, and the polar plane feature Transformer extraction submodule includes the following steps:

[0054] The spatial feature Mamba extraction submodule, the angular feature Mamba extraction submodule, and the polar plane feature Transformer extraction submodule are cascaded and residually connected to obtain the light field feature Tra-Mba extraction submodule, denoted as: The calculation process is as follows:

[0055] ;

[0056] in, and These represent the input and output features of the Tra-Mba light field feature extraction submodule, respectively.

[0057] A deep feature extraction module for the optical field is constructed by cascading and residually connecting several Tra-Mba extraction submodules for optical field features, denoted as: The calculation process is as follows:

[0058] ;

[0059] ;

[0060] in, Indicates input, This represents a function for the deep feature extraction module of the light field. Indicates deep features; Indicates shallow features. This indicates that the operation of the Tra-Mba light field feature extraction submodule has been performed six times.

[0061] Preferably, the image reconstruction module obtains a high-resolution light field image based on the shallow features, the deep features, and the input low-resolution light field image, including the following steps:

[0062] The shallow and deep features are converted into light field fusion features. The calculation process is as follows:

[0063] ;

[0064] ;

[0065] in, and These represent shallow and deep features, respectively. This indicates a channel-level concatenation operation; Indicates a fusion operation;

[0066] Light field fusion characteristics Upsampling is performed to obtain upsampled features. It is expressed as follows:

[0067] ;

[0068] in, The scale factor is represented as or Subpixel convolutional layers; express Convolutional layer;

[0069] Combined with upsampling With high-resolution images The reconstructed high-resolution image is represented as:

[0070] ;

[0071] Among them, high-resolution images It is generated by direct bicubic upsampling of the low-resolution light field image to be reconstructed.

[0072] The present invention has the following beneficial effects:

[0073] (1) This invention designs a state space module for perceiving empty corner features. Unlike existing state space modules, its core lies in a multi-branch state space mechanism that divides the input features into channels and decomposes them into multi-branch parallel processing, thereby achieving refined extraction of empty corner features. At the same time, it is combined with a deep convolution to enhance local details and a difference enhancement module to amplify the effective differences between features, enabling more refined feature extraction. In addition, a designed deep convolution feedforward layer is added to the empty corner feature perceiving state space module. Through convolution, a progressive processing of "dimensionality reduction-deep feature extraction-dimensionality enhancement" is achieved, refining the feature details in each channel with low computational cost.

[0074] (2) The present invention designs a parallax perception self-attention mechanism, which is different from the self-attention mechanism of the prior art. Its core is to add a designed multi-channel perceptron to enhance the feature details in each channel.

[0075] (3) The present invention designs a light field spatial feature Mamba extraction submodule and a light field angle feature Mamba extraction submodule, which simultaneously models the interdependence between features from the spatial and angular dimensions of the light field image, and realizes the full interaction of spatial and angular information.

[0076] (4) The present invention designs a polar plane feature Transformer extraction submodule to effectively extract the polar plane features of the light field and enhance the global disparity feature expression capability of the LFSR model.

[0077] (5) This invention maps 4D light field images into three types of 2D image representations: sub-aperture images (SAI), macro-pixel images (MacPI), and epipolar plane images (EPI). It utilizes a feature extraction module based on a state space mechanism for perceiving empty angle features to fully interact with the spatial and angular information in the light field image, and utilizes a feature extraction module based on a self-attention mechanism for parallax perception to effectively explore the epipolar plane features of the light field image, thereby greatly improving the reconstruction quality of low-resolution light field images.

[0078] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. Attached Figure Description

[0079] Figure 1This is a flowchart of a method according to an embodiment of the present invention;

[0080] Figure 2 This is a schematic diagram of the shallow feature extraction module according to an embodiment of the present invention;

[0081] Figure 3 This is a schematic diagram of the multi-branch state space mechanism according to an embodiment of the present invention;

[0082] Figure 4 This is a schematic diagram of the state space mechanism for sensing empty angle features according to an embodiment of the present invention;

[0083] Figure 5 This is a schematic diagram of a depthwise convolutional feedforward layer according to an embodiment of the present invention;

[0084] Figure 6 This is a schematic diagram of the empty angle feature sensing state space module according to an embodiment of the present invention;

[0085] Figure 7 This is a schematic diagram of the parallax perception self-attention mechanism according to an embodiment of the present invention;

[0086] Figure 8 This is a schematic diagram of the parallax feature perception self-attention module according to an embodiment of the present invention;

[0087] Figure 9 This is a schematic diagram of the spatial feature Mamba extraction submodule according to an embodiment of the present invention;

[0088] Figure 10 This is a schematic diagram of the angle feature Mamba extraction submodule in an embodiment of the present invention;

[0089] Figure 11 This is a schematic diagram of the polar plane feature Transformer extraction submodule according to an embodiment of the present invention;

[0090] Figure 12 This is the Tra-Mba extraction submodule for light field features in this embodiment of the invention;

[0091] Figure 13 This is a schematic diagram of the light field deep feature extraction module according to an embodiment of the present invention;

[0092] Figure 14 This is a schematic diagram of the image reconstruction module according to an embodiment of the present invention;

[0093] Figure 15 This is a schematic diagram of the light field image super-resolution reconstruction network according to an embodiment of the present invention;

[0094] Figure 16 This is a comparison of the reconstruction results of the method in this embodiment of the invention with those of other methods. Detailed Implementation

[0095] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0096] See Figure 1 A method for super-resolution reconstruction of light field images based on hybrid self-attention and state space includes the following steps:

[0097] S101 uses convolution operations to construct a shallow feature extraction module to extract shallow features from the input low-resolution light field image;

[0098] S102, the first basic feature extraction unit is constructed by combining a multi-branch structure and a state space model, and the second basic feature extraction unit is constructed by combining a multi-head sub-attention mechanism and a channel multilayer perceptron.

[0099] S103, construct a spatial feature Mamba extraction submodule and an angle feature Mamba extraction submodule based on the first basic feature extraction unit, which are used to extract spatial features and angle features from the shallow features, respectively; construct a polar plane feature Transformer extraction submodule based on the second basic feature extraction unit, which is used to extract polar plane features from the shallow features;

[0100] S104, a deep feature extraction module is constructed based on the spatial feature Mamba extraction submodule, the angle feature Mamba extraction submodule, and the polar plane feature Transformer extraction submodule, and deep features are extracted based on the shallow features;

[0101] S105, the image reconstruction module obtains a high-resolution light field image based on the shallow features, the deep features, and the input low-resolution light field image;

[0102] S106, combined with a shallow feature extraction module, a deep feature extraction module, and an image reconstruction module, forms a super-resolution network used to achieve super-resolution reconstruction of light field images.

[0103] Specifically, in step S101, the input to the shallow feature extraction module is a sub-aperture image of the low-resolution light field image to be reconstructed. See [link to S101]. Figure 2 As shown, in this step, the size of the angle dimension is obtained. , And the size of the spatial dimension is , Low-resolution light field image to be reconstructed Convert it into a sub-aperture image format As input, R represents the real number field. A shallow feature extraction module is constructed. Used to extract shallow features Its expression is as follows:

[0104] ;

[0105] ;

[0106] in, The sub-aperture image form representing the light field image to be reconstructed. Represents shallow features. Represents the shallow feature extraction module. Indicates the kernel size as Convolutional layers do not change the number of channels in the input feature map. This represents a convolutional layer function that upsamples the number of channels in the input feature map to 64. This represents the LReLU activation function.

[0107] Specifically, in S102, the empty angle feature perception state space unit is constructed using the empty angle feature perception state space mechanism as the first basic feature extraction unit.

[0108] State-space model This includes parameterization and discretization. The parameterization calculation process is as follows:

[0109] ;

[0110] ;

[0111] in, , They represent lengths of The input sequence and the output sequence, Indicates length is The learnable hidden state space This represents the hidden state space of the previous time step. , and Let represent the discretized parameter matrix, and The value is usually 64. It represents a real number.

[0112] Discretization employs the zero-order preservation rule to discretize the continuous parameter matrix, and its calculation process is as follows:

[0113] ;

[0114] ;

[0115] ;

[0116] in, Indicates the time step. , and Represents the discretized parameter matrix.

[0117] Integrating the above formulas yields a discretized state-space model. The calculation process is as follows:

[0118] ;

[0119] ;

[0120] in, Represents the hidden state space at the previous time step and the hidden state space at the current time step. From the hidden state space of the previous moment The calculation yielded the result.

[0121] See Figure 3 As shown, a multi-branch state-space mechanism is constructed using a state-space model. The calculation process includes the following steps:

[0122] Channel division is represented as:

[0123] ;

[0124] The local feature extraction branch is represented as follows:

[0125] ;

[0126] The global feature extraction branch is represented as:

[0127] ;

[0128] ;

[0129] ;

[0130] ;

[0131] Feature fusion, represented as:

[0132] ;

[0133] in, This represents the input to the multi-branch state-space mechanism. This indicates a channel partitioning operation. This represents the intermediate features after channel segmentation. This represents depthwise separable convolution. This represents the global average pooling operation. This represents the output feature after global average pooling. Indicates an upsampling operation. Representing a state-space model, This represents the output features after upsampling and state-space model processing. This represents the features extracted after differential extraction. Global features extracted by multi-branch state-space mechanism Local features extracted by the multi-branch state-space mechanism This indicates that features will be fused by channel. This represents the output feature after fusing global and local features.

[0134] See Figure 4 As shown, a multi-branch state-space mechanism is further utilized to construct a state-space mechanism for perceiving empty angle features. The expression is as follows:

[0135] ;

[0136] Among them, the empty angle perception state space mechanism The input is the layer-normalized input features. The output is the final refined feature obtained; Represents a linear mapping layer. This represents depthwise separable convolution. This represents the activation function. This represents a multi-branch state-space mechanism. This represents the Hadama dot product.

[0137] See Figure 5 As shown, a further constructed deep convolutional feedforward layer The expression is as follows:

[0138] ;

[0139] in, express Convolutional layer This represents a depthwise separable convolution.

[0140] See Figure 6 As shown, the empty angle feature sensing state space unit is finally constructed using the empty angle feature sensing state space mechanism. The calculation process is as follows:

[0141] ;

[0142] ;

[0143] ;

[0144] in, This represents the light field image features of the input empty angle feature perception state space module. This indicates that the empty angle feature perception state space module extracts intermediate features from the light field image. The spatial module for perceiving the spatial features of the empty angle image extracts spatial features from the light field image. Represents the first basic feature extraction unit. The function of the presentation layer is normalized. express Convolutional layers, This indicates a depthwise convolutional feedforward layer. , This represents a learnable residual connection. This represents the state space mechanism for perceiving empty angle features.

[0145] Specifically, in S103, a parallax feature perception self-attention unit is constructed using a parallax perception self-attention mechanism as the second basic feature extraction unit;

[0146] Multi-head self-attention mechanism The calculation process is as follows: First, the light field features of the given sub-aperture image mode are... Remodeling into an embedded sequence The input to the horizontal polar plane Transformer is used to learn distance space dependencies, where C is the number of channels (i.e., the embedding dimension). Subsequently, The process of generating query, key, and value can be represented as:

[0147] ;

[0148] ;

[0149] ;

[0150] This represents three linear projection layers. These are the query, key, and value matrices, respectively. Self-attention is performed on P independent subspaces (multi-head attention counts). Calculate MSA. For the first... subspace , and The embedding dimension is The calculation process is as follows:

[0151] ;

[0152] After concatenating the results from all subspaces and performing a linear projection again, a weighted sequence is obtained. Defined as:

[0153] ;

[0154] in, [⋅] denotes a linear projection layer, and [⋅] denotes a stitching operation. To extract disparity features more precisely, a multilayer perceptron is added. Module, expression as follows:

[0155] ;

[0156] in, Represents a linear mapping layer. This represents depthwise separable convolution. This represents the activation function.

[0157] See Figure 7 As shown, a parallax perception self-attention mechanism is constructed using a multi-head self-attention mechanism. As shown below:

[0158] ;

[0159] ;

[0160] ;

[0161] in, This represents the input of the parallax perception self-attention mechanism. This represents the intermediate features extracted by the parallax perception self-attention mechanism. This represents the output after extracting disparity features. Represents a multi-head self-attention mechanism. The function of the presentation layer is normalized. This refers to a multilayer sensor for the channel.

[0162] See Figure 8 As shown, a parallax feature perception self-attention unit is further constructed using a parallax perception self-attention mechanism, and its calculation process is as follows:

[0163] ;

[0164]

[0165] ;

[0166] in, The light field image features representing the input disparity feature perception self-attention module are represented. This indicates that the parallax feature perception self-attention module extracts intermediate features from the light field image. This indicates that the parallax feature perception self-attention module extracts angular features from the light field image. Represents a self-attention unit for parallax feature perception. express Convolutional layer This represents the activation function. This represents the self-attention mechanism for parallax perception.

[0167] Specifically, in S103, a spatial feature Mamba extraction submodule is constructed based on the first basic feature extraction unit. and angular feature Mamba extraction submodule These are used to extract spatial and angular features of the light field from shallow features, respectively. See also Figures 9 to 10 As shown, it specifically includes the following:

[0168] Constructing a spatial feature Mamba extraction submodule Its transformation unit is based on matrix dimension. First basic feature extraction unit , Convolutional layer The composition and calculation process are as follows:

[0169] ;

[0170] ;

[0171] ;

[0172] in, This represents the input features of the spatial feature extraction submodule of Mamba. This represents the intermediate features extracted from the submodules by Mamba to represent spatial features. This indicates a matrix dimension transformation operation to convert input features into SAI sub-aperture image features. express Convolutional layer The spatial features represented are the output features of the Mamba extraction submodule.

[0173] Constructing an angle feature Mamba extraction submodule It includes matrix dimension transformation units. , First basic feature extraction unit , Convolutional layer The composition and calculation process are as follows:

[0174] ;

[0175] ;

[0176] ;

[0177] in, This represents the input features of the angle feature Mamba extraction submodule. This represents the intermediate features extracted from the Mamba submodule representing angular features. This indicates a matrix dimension transformation operation to convert the sub-aperture image features into macro-pixel image features. This indicates a matrix dimension transformation operation to convert the macro-pixel image features into sub-aperture image features; This represents the state space module for sensing empty angle features. Indicates the Convolutional layer. The angular features represent the output features of the Mamba extraction submodule.

[0178] Specifically, in S103, a polar plane feature Transformer extraction submodule is constructed based on the second basic feature extraction unit to extract polar plane features of the light field from shallow features. See [link to S103]. Figure 11 As shown, it specifically includes the following:

[0179] Constructing a polar plane feature Transformer extraction submodule It includes matrix dimension transformation units. , , , Second basic feature extraction unit and The composition and calculation process are as follows:

[0180] ;

[0181] ;

[0182] ;

[0183] in, This represents the input features of the polar plane feature extraction submodule of the Transformer. This indicates a matrix dimension transformation operation to convert sub-aperture image features into horizontal epipolar plane image features. This represents a matrix dimension transformation operation to convert horizontal epipolar plane image features into sub-aperture image features; This indicates a matrix dimension transformation operation to convert sub-aperture image features into vertical epipolar plane image features. This indicates a matrix dimension transformation operation to convert vertical epipolar plane image features into sub-aperture image features.

[0184] Specifically, in S104, the spatial feature Mamba extraction submodule, the angular feature Mamba extraction submodule, and the polar plane feature Transformer extraction submodule are cascaded and residually connected to obtain the light field feature Tra-Mba extraction submodule. See also Figure 12 As shown, it specifically includes the following:

[0185] ;

[0186] in, This represents the input features of the Tra-Mba light field feature extraction submodule. This indicates the depth features extracted by the Tra-Mba light field feature extraction submodule.

[0187] Specifically, in S104, the deep feature extraction module consists of six Tra-Mba light field feature extraction sub-modules. Constructed by cascaded and parallel residual connections. See also Figure 13 As shown, this step specifically includes the following:

[0188] ;

[0189] ;

[0190] in, Indicates input, This represents a function for the deep feature extraction module of the light field. It represents deep features.

[0191] Specifically, in S105, an image reconstruction module is constructed to fuse shallow and deep features hierarchically. Further, through a fusion feature upsampling unit, a high-resolution image is finally generated from the low-resolution light field image via bicubic upsampling. The images are then superimposed to obtain a fully reconstructed high-resolution light field image. See also... Figure 14 As shown, it specifically includes the following:

[0192] S1051, Construct the sub-feature fusion unit of the image reconstruction module. The function representing the hierarchical feature fusion unit transforms the shallow and deep features into light field fusion features. , means as follows:

[0193] ;

[0194] ;

[0195] in, Indicates the kernel size as Convolutional layer This indicates a channel-level concatenation operation. , These represent the shallow and deep features of the light field, respectively.

[0196] S1052, Construct a fusion feature upsampling unit. The function representing the feature upsampling unit is expressed as follows:

[0197] ;

[0198] in, Indicates the kernel size as Convolutional layer, a function that fuses feature upsampling units. The input is light field fusion features , The scale factor is represented as or Subpixel convolutional layers;

[0199] S1053, cascade the hierarchical feature fusion unit, the fusion feature upsampling unit, and the high-resolution image generated by bicubic upsampling of the low-resolution light field image to be reconstructed. The image reconstruction module constitutes the image reconstruction module. Its function is represented as follows:

[0200] ;

[0201] Specifically, in S106, the shallow feature extraction module, the deep feature extraction module, and the image reconstruction module are cascaded to form a light field image super-resolution reconstruction network, which is used to reconstruct the input low-resolution light field image to be reconstructed into a high-resolution light field image. See [link to relevant documentation]. Figure 15 The calculation process is as follows:

[0202] ;

[0203] in, This represents a low-resolution light field image to be reconstructed. This represents the reconstructed high-resolution light field image. , Indicates the size of the angular dimension. , Indicates the size of spatial dimensions. express or The scale factor, The shallow feature extraction module This refers to the deep feature extraction module of the light field. This refers to the image reconstruction module.

[0204] To better illustrate the advantages of the technical solution of the present invention, the following experiments are disclosed in this embodiment.

[0205] To achieve a comprehensive comparison of quantitative and qualitative performance, this experiment trained and validated the proposed method on five publicly available light field datasets. These datasets contain 144 training scenarios and 23 test scenarios, covering different light field contents and disparity distributions. Specifically, EPFL, HCInew, HCIold, INRIA, and STFgantry were used to construct benchmark datasets. The specific configuration of the training and test sets for the benchmark datasets is detailed in Table 1.

[0206] Table 1 - Training and Test Datasets:

[0207] In the experiments, Bicubic was used as the benchmark, and comparisons were made with the classic deep learning-based single-image super-resolution method RCAN and the current state-of-the-art light field image super-resolution reconstruction methods DistgSSR, LFT, EPIT, LF-DET, and MLFSR. DistgSSR is a CNN-based light field image super-resolution method, LF-DET, LFT, and EPIT are Transformer-based light field image super-resolution methods, and MLFSR is a Mamba-based light field image super-resolution method. The experimental data were obtained from the code and model training provided by the authors of the respective papers. The test dataset was used... The super-resolution results are shown in Table 2. The data in the table are the average PSNR and SSIM of all test light field images in the corresponding dataset. The data in bold are the best data, and the data underlined are the second best data.

[0208] Experimental results show that the method of this invention performs best on all five datasets, and the average PSNR and SSIM are 0.17 dB higher than the current state-of-the-art light field image super-resolution reconstruction method MLFSR.

[0209] Table 2 - Different Algorithms Comparison of average PSNR (dB) / SSIM of super-resolution results:

[0210] Furthermore, the present invention provides solutions for more challenging Qualitative results obtained using different methods on super-resolution tasks. For example... Figure 16 As shown, the reconstruction results of the Origami scene using the method of this invention (i.e., Ours in the figure) and existing advanced light field image super-resolution reconstruction methods are presented. Experimental results show that the method proposed in this invention is better at restoring sharp edges and complex textures, which is something most advanced light field image super-resolution reconstruction methods cannot achieve. For example, in the Origami scene from the HCInew dataset, it can be observed that only the method proposed in this invention accurately restores the edges of the intersecting lines on the surface of the jar, outperforming other methods in avoiding common blurring and artifacts, demonstrating excellent detail restoration capabilities.

[0211] In summary, this experiment, through testing and analysis of real and synthetic light field images from 144 training scenarios and 23 test scenarios, demonstrates that the method of this invention exhibits significant advantages in both subjective visual performance and objective performance. The super-resolution reconstruction of light field images achieves high quality, with good restoration of edge and complex texture details. Furthermore, it achieves high PSNR and SSIM performance with lower computational cost, outperforming the most advanced light field image super-resolution reconstruction methods currently available.

[0212] Therefore, this invention, on the one hand, designs a state space module for perceiving spatial angle features based on a multi-branch state space mechanism as the first basic feature extraction unit. This module achieves multi-branch parallel processing by dividing the input features into channels, and combines global average pooling, a state space model, and a difference enhancement mechanism to achieve refined extraction of global and local features. Furthermore, it incorporates a deep convolutional feedforward layer to refine the feature details within each channel at low computational cost. On the other hand, it designs a disparity feature perceiving self-attention module based on a disparity-perceived self-attention mechanism as the second basic feature extraction unit. This module utilizes a multi-head self-attention mechanism combined with a channel multilayer perceptron to specifically model the epiplanetary features of the light field, effectively capturing global disparity correlations. Thus, through a hybrid architecture of state space model and self-attention mechanism, it achieves comprehensive modeling of the spatial-angle-disparity information of the light field image, significantly improving the reconstruction quality of high-frequency details. These improvements overcome the shortcomings of existing technologies.

[0213] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for super-resolution reconstruction of light field images that combines self-attention and state space, characterized in that, Includes the following steps: A shallow feature extraction module is constructed using convolution operations to extract shallow features from the input low-resolution light field image; The first basic feature extraction unit is constructed by combining a multi-branch structure and a state-space model, and the second basic feature extraction unit is constructed by combining a multi-head sub-attention mechanism and a channel multilayer perceptron. Based on the first basic feature extraction unit, a spatial feature Mamba extraction submodule and an angle feature Mamba extraction submodule are constructed to extract spatial features and angle features from the shallow features, respectively. A polar plane feature Transformer extraction submodule is constructed based on the second basic feature extraction unit to extract polar plane features from the shallow features; A deep feature extraction module is constructed based on the spatial feature Mamba extraction submodule, the angular feature Mamba extraction submodule, and the polar plane feature Transformer extraction submodule; the deep feature extraction module extracts deep features based on the shallow features; An image reconstruction module is constructed to obtain a high-resolution light field image based on the shallow features, the deep features, and the input low-resolution light field image. By combining a shallow feature extraction module, a deep feature extraction module, and an image reconstruction module, a super-resolution network is constructed to achieve super-resolution reconstruction of light field images.

2. The method for super-resolution reconstruction of light field images using a hybrid self-attention and state space approach according to claim 1, characterized in that, The operation of the first basic feature extraction unit is represented as follows: This includes the following operations: ； ； in, , and These represent the input features, intermediate features, and output features of the first basic feature extraction unit, respectively. The function of the presentation layer is normalized. express Convolutional layer; This represents a depthwise convolutional feedforward layer, consisting of sequentially connected... Convolution, depthwise separable convolution and Convolution; , This represents a learnable residual connection. This represents the state space mechanism for perceiving empty angle features.

3. The method for super-resolution reconstruction of light field images using hybrid self-attention and state space as described in claim 2, characterized in that, The empty angle feature perception state space mechanism The expression is as follows: ； in, Represents a linear mapping layer. This represents depthwise separable convolution. This represents the activation function. This represents a multi-branch state-space mechanism. This represents the Hadama dot product; Multi-branch state space mechanism The calculation process includes the following steps: Channel division is represented as: ； The local feature extraction branch is represented as follows: ； The global feature extraction branch is represented as: ； ； ； ； Feature fusion, represented as: ； in, This represents the input to the multi-branch state-space mechanism. This indicates a channel partitioning operation. This represents the intermediate features after channel segmentation; Local features extracted by the multi-branch state-space mechanism; This represents the global average pooling operation. This represents the output feature after global average pooling operation; Indicates an upsampling operation. Representing a state-space model, This represents the output feature after upsampling and state-space model processing; This represents the features extracted after differentiation. Global features extracted using a multi-branch state-space mechanism; This indicates that features will be fused by channel. This represents the output feature after fusing global and local features.

4. The method for super-resolution reconstruction of light field images using a hybrid self-attention and state space approach according to claim 3, characterized in that, The operation of the spatial feature Mamba extraction submodule is represented as follows: The calculation process is as follows: ； ； in, , and These represent the input features, intermediate features, and output features of the spatial feature Mamba extraction submodule, respectively. This represents a matrix dimensionality transformation operation that converts input features into sub-aperture image features. express Convolutional layer.

5. The method for super-resolution reconstruction of light field images using a hybrid self-attention and state space approach according to claim 4, characterized in that, The operation of the angle feature Mamba extraction submodule is represented as follows: The calculation process is as follows: ； ； in, , and These represent the input features, intermediate features, and output features of the angle feature Mamba extraction submodule, respectively. This represents a matrix dimension transformation operation that converts sub-aperture image features into macro-pixel image features. This represents a matrix dimension transformation operation that converts the macro-pixel image features into sub-aperture image features; express Convolutional layer.

6. The method for super-resolution reconstruction of light field images using a hybrid self-attention and state space approach according to claim 5, characterized in that, The operation of the second basic feature extraction unit is represented as follows: This includes the following operations: ； ； in, , and These represent the input features, intermediate features, and output features of the second basic feature extraction unit, respectively. express Convolutional layer This represents the activation function. This represents the self-attention mechanism for parallax perception.

7. The method for super-resolution reconstruction of light field images using a hybrid self-attention and state space approach according to claim 6, characterized in that, The parallax perception self-attention mechanism The calculation process is as follows: ； ； in, , and These represent the input features, intermediate features, and output features of the parallax perception self-attention mechanism, respectively. Represents a multi-head self-attention mechanism. The function of the presentation layer is normalized. This refers to a multilayer sensor for the channel.

8. The method for super-resolution reconstruction of light field images using a hybrid self-attention and state space approach according to claim 7, characterized in that, The polar plane feature Transformer extraction submodule is represented as follows: The calculation process is as follows: ； ； in, , and These represent the input features, intermediate features, and output features of the polar plane feature extraction submodule of the Transformer. This represents a matrix dimension transformation operation that converts sub-aperture image features into horizontal epipolar plane image features. This represents a matrix dimension transformation operation that converts horizontal epipolar plane image features into sub-aperture image features; This represents a matrix dimension transformation operation that converts sub-aperture image features into vertical epipolar plane image features. This represents a matrix dimension transformation operation that converts vertical epipolar plane image features into sub-aperture image features.

9. The method for super-resolution reconstruction of light field images using a hybrid self-attention and state space approach according to claim 8, characterized in that, The deep feature extraction module, constructed based on the spatial feature Mamba extraction submodule, the angular feature Mamba extraction submodule, and the polar plane feature Transformer extraction submodule, includes the following steps: The spatial feature Mamba extraction submodule, the angular feature Mamba extraction submodule, and the polar plane feature Transformer extraction submodule are cascaded and residually connected to obtain the light field feature Tra-Mba extraction submodule, denoted as: The calculation process is as follows: ； in, and These represent the input and output features of the Tra-Mba light field feature extraction submodule, respectively. A deep feature extraction module for the optical field is constructed by cascading and residually connecting several Tra-Mba extraction submodules for optical field features, denoted as: The calculation process is as follows: ； ； in, Indicates input, This represents a function for the deep feature extraction module of the light field. Indicates deep features; Indicates shallow features. This indicates that the operation of the Tra-Mba light field feature extraction submodule has been performed six times.

10. The method for super-resolution reconstruction of light field images using a hybrid self-attention and state space approach according to claim 1, characterized in that, The image reconstruction module obtains a high-resolution light field image based on the shallow features, the deep features, and the input low-resolution light field image, including the following steps: The shallow and deep features are converted into light field fusion features. The calculation process is as follows: ； ； in, and These represent shallow features and deep features, respectively. This indicates a channel-level concatenation operation; Indicates a fusion operation; Light field fusion characteristics Upsampling is performed to obtain upsampled features. It is expressed as follows: ； in, The scale factor is represented as or Subpixel convolutional layers; express Convolutional layer; Combined with upsampling With high-resolution images The reconstructed high-resolution image is represented as: ； Among them, high-resolution images It is generated by direct bicubic upsampling of the low-resolution light field image to be reconstructed.

Images