A deep geometry prior guided mamba light field angular super-resolution method

Abstract

A deep geometry-prior-guided Mamba light field angle super-resolution method, belonging to the field of computer vision, is proposed. It estimates a high-precision disparity map using PSV; employs a dual-branch parallel approach to extract image texture features and deep structural features separately, and calculates adaptive weights for cross-modal features using a gated attention mechanism, outputting geometrically enhanced fused features; utilizes a CNN branch to extract detailed residuals of local spatial texture, and uses a Mamba branch to perform global modeling based on the geometrically enhanced features for long-range angular dimension dependence; and dynamically adjusts the ratio of local details to global angular information according to scene content using a channel attention mechanism to achieve an adaptive balance between local spatial fidelity and global angular consistency; and employs an upsampling module to map implicit empty angle features to a dense angular domain, outputting high-quality light field angle super-resolution results. This invention effectively alleviates the blurring problem in image reconstruction and achieves adaptive feature fusion, ensuring the unity of spatial consistency and angular continuity.

Description

Technical Field

[0001] This application belongs to the field of computer vision, specifically relating to a depth geometry prior-guided Mamba light field angle super-resolution method. Background Technology

[0002] With the development of computational imaging technology and artificial intelligence, human needs for 3D information acquisition and realistic reconstruction are constantly increasing. Unlike traditional imaging methods that only record the intensity and position of light in space, light field cameras, by inserting a microlens array between the main lens and the sensor, can simultaneously capture the spatial position and direction information of light, theoretically providing a complete description of the light distribution in the 3D world. This enables advanced functions such as image capture followed by refocusing and multi-view reconstruction. As a novel imaging technology, light field imaging has shown great potential in computer vision, 3D reconstruction, and augmented reality in recent years. However, due to the spatial-bandwidth product limitation of optical systems in terms of imaging field of view and resolution, light field cameras inevitably sacrifice spatial resolution while improving angular resolution, resulting in lower quality sub-aperture images. This trade-off between spatial and angular bandwidth limits the application of light field imaging in scenarios requiring high angular resolution.

[0003] Angular Super-Resolution (ASR) aims to recover high-density angular information from sparse multi-view observations to achieve realistic scene reconstruction. Existing methods can be broadly categorized into two types: one is depth-guided reconstruction methods that rely on explicit geometric information. These methods estimate scene depth or parallax to achieve geometric alignment and pixel reprojection between different viewpoints, thereby improving angular consistency. These methods exhibit good geometric constraint capabilities when handling wide baselines or complex occluded scenes, but are highly sensitive to depth estimation accuracy and susceptible to noise and depth errors. The other type is depth-independent, learning-based feature reconstruction methods. These typically utilize convolutional neural networks (CNNs) or Transformers to directly extract high-dimensional features from light field images and predict missing information. These methods excel in texture restoration and feature representation, but due to the lack of explicit geometric modeling, they are prone to structural blurring or misalignment problems in scenes with large parallax.

[0004] Depth-based methods typically rely on explicit or implicit geometric modeling. Explicit depth-based methods establish correspondences between views by estimating disparity maps, projecting source views onto target views based on multi-view geometry, and then using a learning module to repair residuals and holes generated by the projection. These methods offer strong geometric interpretability, explicitly guaranteeing disparity consistency and effectively handling long-distance correspondence problems in cases of large disparity; however, their super-resolution quality is highly dependent on the accuracy of depth estimation, and estimation errors can easily lead to reconstruction artifacts. Jin et al. constructed an end-to-end network including depth estimation and light field fusion modules and designed an EPI gradient loss to constrain the disparity structure, thereby achieving high-quality angular super-resolution in wide-baseline scenes. DA²N introduces classical frequency domain reconstruction ideas into the image domain, designing learnable modules and anti-aliasing structures to simultaneously address large disparity and non-Lambertian surface problems. Geo-NI integrates neural interpolation and depth image rendering approaches, generating candidate reconstructions through multiple depth assumptions and using cost volumes to guide fusion, combining geometric constraints with adaptability to non-Lambertian effects.

[0005] Feature-based learning methods focus on learning the intrinsic feature representations between angles from data without explicitly relying on depth information. LF-InterNet designs spatial and angular feature extractors to decouple feature extraction and progressive fusion of macro-pixel images, achieving efficient information utilization. DistgASR further introduces an EPI feature extraction module, decoupling spatial, angular, and disparity information at the feature level, modeling them separately and then fusing them to improve angular consistency while preserving spatial details. ViewFormer proposes a view-specific query mechanism that adaptively calculates the offset between the target view and the input view by estimating the disparity map, and then uses sampling strategies such as deformable convolution to achieve disparity-aware feature alignment. GLGNet addresses the problems of large baselines and occlusion by proposing a depth-independent angle super-resolution architecture that preserves the disparity structure through global-local-global feature aggregation and bilateral upsampling, and predicts interpolation weights based on sub-pixel offsets and range differences.

[0006] In the field of light field angle super-resolution, we still face the dual challenges of insufficient high-frequency detail restoration and poor angle consistency. On the one hand, although deep neural networks have made significant progress in light field reconstruction, their feature extraction and geometric modeling capabilities in complex scenes remain limited. Pure CNN solutions are limited by local receptive fields and cannot effectively model long-range angle dependencies across viewpoints, resulting in disparity jumps and poor angle consistency in the generated viewpoint. Transformer-type solutions can achieve global modeling, but the computational complexity of self-attention increases quadratically with angular resolution, leading to extremely low inference efficiency in high-angle super-resolution scenarios. Existing Mamba-based light field solutions mostly focus on spatial super-resolution tasks and have not fully explored their linear complexity advantage in modeling cross-viewpoint dependencies in the light field angle dimension, and they also suffer from insufficient restoration of local fine textures due to causal scanning. Existing CNN and Mamba fusion schemes mostly employ a serial cascaded structure, which cannot simultaneously handle the refined extraction of local high-frequency textures and the parallel modeling of global angle dependencies. Furthermore, dual-branch feature fusion often uses fixed-weight channel concatenation or addition strategies, failing to adaptively allocate feature weights based on the texture complexity and parallax variation of scene regions. This results in over-smoothing of flat areas and blurring of details and artifacts in edge-occluded areas. In light field data, spatial and angular dimensions are highly coupled, especially in the presence of large parallax and complex occlusion. The network struggles to simultaneously maintain spatial detail and angular consistency, often leading to texture blurring and edge distortion. On the other hand, in wide-area light fields, due to the wide baseline and large parallax range, the ray projection relationships between different viewpoints are more complex, particularly in areas with drastic depth changes or occlusion boundaries, making it difficult for the network to learn continuous and stable angular relationships. In existing solutions, disparity / depth information is often used only as a preprocessing step, without being deeply coupled with the subsequent angle super-resolution process. Furthermore, the fusion of depth information and texture features often employs simple feature stitching, failing to achieve full guidance of the feature extraction process from geometric priors. This results in a mismatch between the texture details of the generated viewpoint and the 3D geometry of the scene, making it difficult to guarantee disparity continuity and angle consistency. Therefore, effectively addressing these issues remains an important research direction in the field of light field angle super-resolution. Summary of the Invention

[0007] To address the shortcomings of existing technologies in high-frequency detail reproduction and poor angular consistency, this application proposes a depth geometry prior-guided Mamba light field angular super-resolution method, belonging to the field of computer vision. It utilizes Plane-Sweep Volume (PSV) to construct a geometric structure prior for the scene and injects depth information as a structural constraint into the angular feature learning process through a gated attention mechanism, thereby overcoming the deficiency of loose coupling between texture and geometric features in traditional methods. A geometry-texture collaborative feature extraction framework is constructed: a PSV is built based on the input sub-aperture array to estimate a high-precision disparity map, which serves as the geometric anchor point for subsequent empty angle modeling; a dual-branch parallel encoding strategy is adopted to extract image texture features and deep structural features respectively, and an adaptive weight of cross-modal features is calculated through a gated attention mechanism to achieve selective enhancement of texture information under the guidance of geometric structure, outputting geometrically enhanced fusion features; a multi-branch architecture is designed, using a CNN branch to extract the detail residuals of local spatial texture, and a Mamba branch to perform global modeling based on geometrically enhanced features for long-range angular dimension dependence, and a channel attention mechanism to dynamically adjust the contribution ratio of local details and global angular information according to scene content, achieving an adaptive balance between local spatial fidelity and global angular consistency; finally, a PixelShuffle 2D structure is used to map implicit empty angle features to the dense angular domain, outputting high-quality light field angular super-resolution results.

[0008] A deep geometry-prior-guided Mamba optical field angle super-resolution method specifically includes the following steps:

[0009] S1, Depth Estimation:

[0010] A PSV is constructed based on sparse input viewpoint images. The disparity map corresponding to the input viewpoint image is obtained through a depth estimation module, which serves as a priori geometric structure of the scene.

[0011] The depth estimation module adopts a hierarchical, progressive convolutional architecture;

[0012] S2, cross-modal adaptive fusion:

[0013] The input viewpoint image and disparity map are encoded in a dual-branch parallel manner through the feature fusion module, and the adaptive weights of the encoded features are calculated to output a fused feature representation with depth perception characteristics.

[0014] The feature fusion module includes an image convolution module, a depth convolution module, a channel gating module, and a spatial gating module;

[0015] S3, Joint Feature Extraction of Empty Angles:

[0016] A feature extraction model is used to perform multi-level deep feature extraction on the fused feature representation, realizing the layer-by-layer reinforcement and propagation of geometric structure information in the deep network, and outputting joint features of void angles;

[0017] The feature extraction model adopts a progressive enhancement architecture of cascaded grouping and hierarchical feature groups;

[0018] S4, angle upsampling:

[0019] The upsampling module performs angular upsampling on the joint features of the empty angle to output a high-resolution light field image.

[0020] The upsampling module adopts a PixelShuffle 2D structure;

[0021] S5, Joint Optimization:

[0022] End-to-end optimization of the depth estimation model and the feature extraction model is performed using a joint loss function;

[0023] The joint loss function includes image reconstruction loss and depth consistency constraint, which is used to update network parameters to improve angle super-resolution quality and geometric accuracy.

[0024] Furthermore, in S1, the process of obtaining the disparity map corresponding to the input viewpoint image is as follows:

[0025] S11, PSV Construction: Constructing the PSV at the input viewpoint based on the sparse input viewpoint image:

[0026] First, the rearrangement operation: rearrange the input viewpoint image. Sub-Aperture Image (SAI) is reconstructed from a sub-aperture image array format. ,in For the set of real numbers, For batch size, The input angular dimension is h, and the width and height of a single sub-aperture image are h and w, respectively.

[0027] Then, a reverse deformation construction mechanism is used to construct the sub-aperture image. Corresponding PSV: Based on preset Discrete disparity planes By reprojecting sub-aperture images , will input Arbitrary sampling positions and the synthesized target positions are simultaneously encoded into the PSV to generate a set of deformed images corresponding to each disparity plane. Sub-aperture image The corresponding PSV; among which, The number of discrete disparity planes. For a single disparity plane, The images obtained after reprojection and distortion of each parallax plane;

[0028] S12, Disparity Regression: Based on Sub-Aperture Images The corresponding PSV, combined with multi-view stereo vision constraints, estimates the disparity map of the input viewpoint. ;

[0029] The parallax regression module performs feature learning and information aggregation from three dimensions: intra-view, inter-depth, and spatial context.

[0030] The disparity regression module adopts a hierarchical and progressive convolutional architecture, which includes a plane-by-plane convolution module, a cross-plane convolution module, and a disparity prediction module.

[0031] First, the plane-wise convolution module reconstructs the PSV obtained in S11 into... The two-dimensional feature mapping is used to learn view-invariant feature representations by independently fusing multi-view information on each depth plane through four cascaded convolutional layers. Then, the cross-plane convolutional module shifts the focus to consistency modeling of the depth dimension by flattening the depth dimension into a channel dimension and learning continuity constraints in the depth hypothesis space using the equivalent form of three-dimensional convolution. Finally, the disparity prediction module concatenates all features from each disparity plane and predicts disparity values ​​through consecutive convolutional layers, outputting a disparity map. .

[0032] Furthermore, in S2, the cross-modal adaptive fusion process is as follows:

[0033] The input viewpoint image and disparity map are used as inputs to the feature fusion module. The image convolution module and the depth convolution module perform dual-branch parallel encoding. Then, the channel gating module and the spatial gating module calculate the adaptive weights of the encoded features, thereby realizing the selective enhancement of texture features under the guidance of geometric structure and outputting a fusion feature representation with depth perception characteristics.

[0034] S21, input viewpoint image Parallax map As input, the apparent texture features are obtained through dual-branch parallel encoding using an image convolution module and a depth convolution module. with depth features ;

[0035] The image convolution module is used to preserve the texture details and color fidelity of the image, for the input viewpoint image. Two-stage convolutional coding is performed; the image convolution module uses two convolutional layers with a kernel size of 3×3 and a stride of 1. ReLU activation is used between the convolutional layers for non-linear activation, outputting apparent texture features. ;

[0036] The deep convolution module is used to extract the geometric structure and spatial relationships of the scene, and to process the disparity map. Three-level convolutional encoding is performed; the depthwise convolutional module uses three convolutional layers with a kernel size of 3×3 and a stride of 1. ReLU activation is used between convolutional layers for non-linear activation, outputting depth features. ;

[0037] S22, Calculate the disparity map mean and standard deviation The normalized disparity map is obtained by normalizing the data. The Sobel operator is used to compute the normalized disparity map. The horizontal and vertical gradients are used to obtain the depth geometric structure features. ;

[0038] For deep geometric features Perform edge enhancement: enhance apparent texture features Depth features Normalized disparity map and depth geometric features The features are concatenated along the channel dimension and fed into a feature fusion convolutional network to output geometrically enhanced features with reliable geometric priors. ;

[0039] The Sobel operator includes and ; and The convolution kernels are all 3×3;

[0040] , ;

[0041] The feature fusion convolutional network includes two convolutional layers, each with a kernel size of 3×3 and a stride of 1. The ReLU activation function is used between the convolutional layers.

[0042] S23, Appearance texture features based on gating attention mechanism and depth features Perform adaptive fusion to output gated weighted geometric enhancement features and fusion weights;

[0043] First, global mean pooling is used to enhance the geometric features. Pooling as a geometric enhancement feature Then, after passing through the channel gating module, the channel attention weights are output. ;

[0044] The channel gating module includes cascaded channel compression convolution and channel expansion convolution; the activation function of the channel compression convolution is the ReLU activation function, and the kernel size is 1×1; the activation function of the channel expansion convolution is the Sigmoid activation function, and the kernel size is 1×1.

[0045] Then, pooling geometric enhancement features Normalized disparity map and depth geometric features The data is concatenated along the channel dimension, input to the spatial gating module, and output as spatial attention weights. ;

[0046] The spatial gating module includes two cascaded convolutional layers; the first convolutional layer has a kernel size of 3×3 and uses the ReLU activation function; the second convolutional layer has a kernel size of 1×1 and uses the Sigmoid activation function.

[0047] The pooled geometric augmentation features and channel attention weights after channel gating are combined. After multiplication, it is then combined with the spatial attention weight. The layers are concatenated along the channel dimension; the concatenated result is then input into a convolutional layer to obtain the fusion weights. ;

[0048] The convolutional layer has a kernel size of 1×1 and uses the Sigmoid activation function.

[0049] S24, pooling geometry enhancement features With fusion weight The enhancement is obtained by multiplication; then residual fusion is applied to the input viewpoint image. The summation outputs a fused feature representation with depth-aware characteristics. .

[0050] Furthermore, in S3, the process of extracting the joint feature of the empty angle is as follows:

[0051] The progressive enhancement architecture includes four sets of cascaded feature modules, each set of cascaded modules includes four sets of hierarchical feature modules, and each hierarchical feature module is a CNN-Mamba multi-branch feature extraction block;

[0052] The CNN-Mamba multi-branch feature extraction block includes an angular dimension geometric perception aggregation submodule, a depth-constrained EPI structure consistency extraction submodule, a spatial high-frequency detail preservation submodule, a geometric edge response submodule, and a geometrically enhanced sequence state space global modeling submodule.

[0053] S31, fusing feature representations through geometrically guided CNN-Mamba multi-branch feature extraction blocks. Parallel extraction of multi-dimensional spatial features:

[0054] S311, the angle dimension geometric perception aggregation submodule aggregates the correlation between different viewpoints of the light field under the prior constraints of geometric structure;

[0055] The angle-dimensional geometry-aware aggregation submodule sequentially includes a non-filled convolutional layer and a convolutional layer; the kernel of the non-filled convolutional layer is α×α, and the stride is α; the kernel size of the convolutional layer is... The number of output channels is 64× ;

[0056] First, padding-free convolutions are used to represent the fused features. Angular dimension structured downsampling is performed to obtain feature maps, thereby achieving the aggregation of geometric consistency information within the local viewpoint neighborhood;

[0057] Then, the features are mapped to a high-dimensional angle space through a convolutional layer to obtain the channel-dimensional angle encoding information;

[0058] Finally, the pixel shrug operation is used to rearrange the angle encoding information from the channel dimension to the spatial dimension, achieving the synergistic integration of geometrically perceived angle features and spatial dimensions, and outputting angle-enhanced features. ;

[0059] S312, guided by depth geometry priors, the depth-constrained EPI structure consistency extraction submodule extracts the structure consistency features of the horizontal epipolar (EPI) and vertical epipolar directions of the light field respectively; through the constraint of the epipolar geometry by the depth priors, the structure consistency of the light field in occluded areas and weak texture areas is effectively enhanced.

[0060] The depth-constrained EPI structure consistency extraction submodule includes two parallel EPI branches: a horizontal EPI branch and a vertical EPI branch.

[0061] The horizontal EPI branch first passes through a convolution kernel with a size of 1× Step size is Fill as Convolutional layers for fused feature representation A sliding encoding along the horizontal viewpoint direction is performed to capture the continuity of the horizontal epipolar structure under depth prior constraints; then, after dimensionality upscaling via 1×1 convolution, one-dimensional pixel rearrangement is used to output the high-dimensional EPI features along the horizontal viewpoint direction. ;

[0062] The vertical EPI branch will fuse feature representations. After performing spatial dimension swapping (i.e., H and W interchange), and executing the same processing flow as the horizontal EPI branch, the original dimensions are restored, and the EPI structural features in the vertical view direction are extracted. ;

[0063] S313, based on geometric enhancement features, extracts high-frequency detail residuals in the local space using the spatial high-frequency detail preservation submodule, which then forms a spatial-angular complementarity with the angular dimension and EPI branch:

[0064] The spatial high-frequency detail preservation submodule employs a double-dilated convolutional structure. By expanding the receptive field, it captures neighborhood spatial context information constrained by geometric structure. Combining the ReLU activation function with a 3×3 spatial feature enhancement convolution, it selectively enhances geometric boundaries and high-frequency texture details, outputting spatial detail enhancement features. ;

[0065] The porosity of the double-hole convolutional structure is... Fill with ;

[0066] S314, the geometric edge response submodule extracts high-frequency responses from edges, structural lines and depth discontinuities, strengthening the constraint effect of geometric priors on empty corner modeling;

[0067] The geometric edge response submodule includes three types of high-frequency operators with fixed convolution kernels. , and First, the three types of high-frequency operators respectively represent the fusion features. Each channel is independently subjected to grouped convolution to obtain three types of high-frequency features; these three types of high-frequency features include horizontal gradient features, vertical gradient features, and second-order high-frequency texture features; then, the three types of high-frequency features are concatenated along the channel dimension and a convolution kernel of size [size missing] is applied. Output channels are Convolution is used to compress features and obtain high-frequency features. ;

[0068] , , ;

[0069] S315, the geometry-enhanced sequence state space global modeling submodule adopts the Mamba architecture for fused feature representation. Long-program sequence dependency modeling is performed, treating the features of each viewpoint of the light field as sequence elements. Leveraging the global receptive field advantage of the State Space Model (SSM), global angular correlations across viewpoints are captured under geometric prior constraints, outputting globally consistent features. ;

[0070] The Mamba architecture is used to represent the fusion features. The specific process of performing long-program column dependency modeling is as follows:

[0071] First, the fusion feature representation Perform multi-directional rearrangement: merge feature representations According to the preset space-angle division rules, each direction is expanded into at least (or greater than) four direction sequences, forming a set of direction sequences. The first direction sequence is achieved by adjusting the input tensor according to... Spatial combination rearrangement And flattened to obtain; the second direction sequence is obtained by pressing the input tensor according to Rearranged as The sequence is obtained by flattening the sequence; the third direction sequence is obtained by reversing the first direction sequence; the fourth direction sequence is obtained by reversing the second direction sequence.

[0072] in, For general sequence variables, This is used to indicate the sequence direction; u and v represent the number of horizontal and vertical input viewpoints, respectively.

[0073] Then, sequence projection transformation: for the set of direction sequences Linear projections are performed in each direction to obtain the time step parameter sequence. State input sequence and output weight sequence The three sets of projection features;

[0074] in, For discrete time steps, For time step parameter sequence, For SSM standard notation—input matrix, For SSM standard notation—output matrix;

[0075] Next, selectively scan the state update: update the set of direction sequences. Time step parameter sequence State decay matrix State input sequence and output weight sequence The input is fed into the selective scan module, where a directional output sequence is generated using a recursive method based on the sequence state space of the decay kernel. ;

[0076] The selective scan module achieves differentiable forward and backward propagation through CUDA core operations.

[0077] Next, multi-directional feature merging: directional output sequences are then processed. Perform a weighted summation in each direction, and then use an inverse transformation to restore the weighted summation result to the three-dimensional feature tensor y corresponding to the input size;

[0078] Finally, feature normalization: for the three-dimensional feature tensor Perform normalization processing to output globally consistent features. ;

[0079] S32 enhances the angle feature output by S311. High-dimensional EPI features in the horizontal view direction output by S312 EPI structural features in the vertical viewing direction output by S312 S313 output spatial detail enhancement features High-frequency characteristics of S314 output and the global consistency characteristics of S315 output The features are concatenated along the channel dimension; then, a channel attention mechanism is used to weight the concatenated multi-branch fusion features channel by channel, dynamically adjusting the contribution ratio of local high-frequency details and global angular consistency; finally, a 1×1 convolution is used for channel compression to output the joint features of empty and angled regions. This is used for subsequent angle upsampling.

[0080] Furthermore, in S4, the angle upsampling process includes:

[0081] The upsampling module includes a first-layer channel-compressed convolution, a second-layer channel-expanded convolution, a third-layer PixelShuffle layer, and a fourth-layer channel-compressed convolution; the kernel size of the first-layer channel-compressed convolution is... Step size is The activation function is ReLU; the kernel size of the second-layer channel-expanded convolution is... The number of channels is 64× The rearrangement factor of the third PixelShuffle layer is: The fourth layer channel compression convolution has a kernel size of 3×3 and an inflation rate of 100%. Fill with ;in, To output the angle dimension;

[0082] S41, using the upsampling module to perform joint features of the empty angle. Angular upsampling is performed to output a high-resolution light field image in macro image format. ;

[0083] S42, high-resolution light field image Convert the image format to sub-aperture format. .

[0084] Furthermore, in S5, the joint optimization process is as follows:

[0085] Computational sub-aperture format image and the image of the viewpoint to be tested loss function between , obtain the loss value Among them, the loss function Including absolute value loss function EPI Consistency Loss Function ;

[0086] The loss value Adam is used as a supervisory term for the depth estimation and feature extraction modules; it is selected as the optimizer to adjust the weights and biases of the depth estimation and feature extraction modules to minimize the loss value. ;

[0087] First, the loss value is calculated using the backpropagation algorithm. For each weight and bias, the gradient;

[0088] Then, the parameters are updated using gradient descent, the training data is input into the light field angle super-resolution network, and the predicted value is calculated through forward propagation;

[0089] The light field angle super-resolution network is an overall network structure composed of a depth estimation module, a feature fusion module, a feature extraction model, and an upsampling module.

[0090] Next, the parameters are updated through backpropagation, and the process is iterated until the light field angle super-resolution network converges. The number of iterations is 60 cycles, and all data in the training set are trained once as one cycle. After training, the trained depth estimation module and feature extraction module are obtained.

[0091] The loss function The calculation is as follows:

[0092] ;

[0093] in, This represents the absolute value loss between the predicted image and the image to be tested. The viewpoint consistency loss between the predicted image and the test image is quantified by calculating the slope loss of the horizontal and vertical EPIs. and They are respectively and The proportion of the loss function.

[0094] An optical field angle super-resolution network utilizing an optical field angle super-resolution method comprises, in sequence, a depth estimation module, a feature fusion module, a feature extraction model, and an upsampling module. The depth estimation module is used to acquire a disparity map corresponding to an input viewpoint image. The feature fusion module is used to encode and fuse the disparity map with the input viewpoint image to obtain a fused feature representation. The feature extraction model is used to perform multi-level depth feature extraction on the fused feature representation to obtain joint features of space and angle. The upsampling module is used to perform angle upsampling on the joint features of space and angle to obtain a high-resolution optical field image.

[0095] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0096] (1) When fusing depth features and image features, this invention designs a multi-layer gated attention network adaptive fusion structure for apparent texture features and depth features. Without losing image features, it effectively integrates depth features for depth guidance, thereby improving the model's representation ability in wide baseline scenes and complex occlusion scenes.

[0097] (2) In the feature extraction stage, the present invention uses PixelShuffle2D to extend the SS2D module to the angle dimension and designs a CNN-Mamba multi-branch feature extraction architecture. The CNN branch captures local spatial features and high-frequency details, and the Mamba branch models global long-range dependencies. The dynamic weighted fusion of the two-branch features is achieved through the channel attention mechanism, which realizes the effective extraction of light field features and effectively alleviates the problem of image reconstruction blurring in traditional methods.

[0098] (3) In the feature fusion stage, the present invention introduces a channel attention mechanism, which dynamically evaluates the importance weight of each feature channel to achieve adaptive feature fusion, thereby maximizing the reconstruction quality and ensuring the unity of spatial consistency and angular continuity. Attached Figure Description

[0099] Figure 1 This is a flowchart illustrating the overall process of the Mamba optical field angle super-resolution method that integrates depth information according to the present invention.

[0100] Figure 2 A comparison of reconstruction results and errors of super-resolution methods with different light field angles in a real scene at a 3_3 viewpoint.

[0101] Figure 2 (a) shows the error mapping between the predicted image and the ground truth image of the DistgASR method, PSNR=30.61dB, SSIM=0.9771; Figure 2 (b) is the predicted image of the DistgASR method in a real scene at a 3_3 viewpoint; Figure 2(c) is the error mapping between the predicted image and the ground truth image of the ELFR method, PSNR=27.81dB, SSIM=0.9597; Figure 2 (d) is the predicted image of the ELFR method in a real scene at a 3_3 viewpoint; Figure 2 (e) is the error mapping between the predicted image and the ground truth image of the EASR method, PSNR=30.73dB, SSIM=0.9752; Figure 2 (f) is the predicted image of the EASR method in a real scene at a 3_3 viewpoint; Figure 2 (g) is the error mapping diagram between the predicted image and the ground truth image of the method of the present invention, PSNR=30.77dB, SSIM=0.9765; Figure 2 (h) is the predicted image of the method of the present invention in a real scene at a 3_3 viewpoint;

[0102] Figure 3 Figure 3_3 shows the reconstruction results and error analysis of super-resolution methods with different light field angles in the Occlusions scene from the viewpoint.

[0103] Figure 3 (a) is the ground truth image of the Occlusions scene from viewpoint 3_3, with the red box representing the subsequent local magnification and comparison area; Figure 3 (b) is the predicted image of the method of the present invention; Figure 3 (c) is its corresponding error map, PSNR=38.83dB, SSIM=0.9761; Figure 3 (d) is a magnified view of the error mapping between the predicted image and the ground truth image of the DistgASR method, PSNR=33.34dB, SSIM=0.9816; Figure 3 (e) is a magnified view of the predicted image of the DistgASR method in a real scene at a 3_3 viewpoint; Figure 3 (f) is a magnified view of the error mapping between the predicted image and the ground truth image using the ELFR method, with PSNR=30.66dB and SSIM=0.9699; Figure 3 (g) is a magnified view of the predicted image of the real scene from a 3_3 viewpoint using the ELFR method; Figure 3 (h) is a magnified view of the error mapping between the predicted image and the ground truth image of the EASR method, PSNR=32.55dB, SSIM=0.9753; Figure 3 (i) is a magnified view of the predicted image of the EASR method in a real scene at a 3_3 viewpoint; Figure 3 (j) is a magnified view of the error mapping between the predicted image and the ground truth image, PSNR=34.16dB, SSIM=0.9822; Figure 3 (k) is a magnified view of the predicted image in the real scene from the 3_3 viewpoint;

[0104] Figure 4 Figure 3_3 shows the reconstruction results and error analysis of super-resolution methods with different light field angles in HCI scenes from different viewpoints.

[0105] Figure 4 (a) is the ground truth image of the HCI scene from the 3_3 viewpoint, with the red box representing the subsequent local magnification and comparison area; Figure 4 (b) is the predicted image of the method of the present invention; Figure 4 (c) is the corresponding error plot, PSNR=32.54dB, SSIM=0.8899; Figure 4 (d) is a magnified view of the error mapping between the predicted image and the ground truth image of the DistgASR method, PSNR=21.6dB, SSIM=0.6248; Figure 4 (e) is a magnified view of the predicted image of the DistgASR method in a real scene at a 3_3 viewpoint; Figure 4 (f) is a magnified view of the error mapping between the predicted image and the ground truth image using the ELFR method, with PSNR=29.69dB and SSIM=0.9246; Figure 4 (g) is a magnified view of the predicted image of the real scene from a 3_3 viewpoint using the ELFR method; Figure 4 (h) is a magnified view of the error mapping between the predicted image and the ground truth image of the EASR method, PSNR=23.05dB, SSIM=0.6996; Figure 4 (i) is a magnified view of the predicted image of the EASR method in a real scene at a 3_3 viewpoint; Figure 4 (j) is a magnified view of the error mapping between the predicted image and the ground truth image in the method of the present invention, PSNR=29.09dB, SSIM=0.9137; Figure 4 (k) is a magnified view of the predicted image of the real scene 3_3 by the method of the present invention. Detailed Implementation

[0106] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0107] A deep geometry prior-guided Mamba optical field angle super-resolution method, such as Figure 1 As shown, the specific steps include:

[0108] S1, Depth estimation step: Construct a planar scan volume (PSV) based on sparse input viewpoint images, estimate the corresponding disparity map, and use it as a priori geometric structure of the scene;

[0109] S2, Cross-modal adaptive fusion step: The input viewpoint image and the disparity map are encoded in a dual-branch parallel manner. The adaptive weights of the cross-modal features are calculated through a gated attention mechanism to achieve selective enhancement of texture features under the guidance of geometric structure and output depth-aware geometric enhancement features.

[0110] S3, Joint feature extraction steps for empty angles: Construct a CNN-Mamba multi-branch feature extraction architecture, use CNN branches to extract local spatial texture detail residuals in the geometric enhancement features, use Mamba branches to perform global modeling of long-range angular dimension dependence of the geometric enhancement features, and dynamically fuse the outputs of the two branches through a channel attention mechanism to achieve an adaptive balance between local spatial fidelity and global angular consistency, and output high-dimensional empty angle features.

[0111] S4, Angle Upsampling Step: The high-dimensional empty angle features are upsampled in the angle dimension using the PixelShuffle 2D structure, mapped to the dense angle domain, and output a dense viewpoint image of the light field;

[0112] S5, Joint optimization step: The depth estimation model and the feature extraction model are optimized end-to-end through a joint loss function. The joint loss function includes image reconstruction loss and depth consistency constraint, which is used to update network parameters to improve the angle super-resolution quality and geometric structure accuracy.

[0113] To verify the performance of the network structure of this invention, specific experiments are conducted below to compare it with existing methods from both qualitative and quantitative perspectives, verifying the performance of the light field angle super-resolution network of this invention. This invention uses two virtual synthetic scene datasets, HCI and HCIold, and two real-world scene datasets, 30scenes and STFlytro, for training. Specifically, the training set uses 20 synthetic scenes and 100 real scenes. The test set uses 9 synthetic scene datasets and 70 real scene datasets. The synthetic scene datasets are from the HCI and HCI old datasets, and the real scene datasets are from the 30scenes, Occlusions, and Reflective datasets. All real-world datasets were captured using a Lytro Illum camera. These datasets cover several important factors for evaluating LF reconstruction methods. Specifically, the synthetic datasets contain high-resolution textures to measure the ability to preserve high-frequency details. The real-world datasets can evaluate the performance of different methods under natural lighting and actual camera distortion. Furthermore, the HCI dataset contains scenes with large parallax, emphasizing the robustness of the algorithm under sparser sampling. The same preprocessing measures are applied to both the training and test sets: selecting the center of each scene in the dataset. The view is used for training and testing.

[0114] Table 1. Five datasets used in the experiment

[0115]

[0116] When quantitatively evaluating network performance, PSNR and SSIM metrics are used to calculate the super-resolution score of the Y-channel optical field image angle. The super-resolution score for each scene is obtained by averaging the scores of all generated sub-aperture images in that scene, and the super-resolution score for each dataset is obtained by averaging the scores of all scenes in that dataset.

[0117] This invention selected four representative algorithms in the field of optical field angle super-resolution (DistgASR, EASR, ELFR, and FS-GAF) for comparative analysis. For quantitative analysis, the PSNR values ​​of different algorithms are shown in Table 1. To highlight performance differences, bold text indicates the optimal value for each scene, and underlined text indicates the second-best value. As shown in the table, compared with the comparative methods, this invention achieves better results in both synthetic and real-world scenes.

[0118] In terms of qualitative analysis: such as Figure 2 As shown, this method can generate images with small errors, clear edges, and good reconstruction results in real-world scenarios. Figure 3 As shown, in the occlusion scene of the Occlusions dataset, compared with the large number of high error regions in the error maps of methods such as DistgASR and EASR, this method, through enhanced long-range information modeling capabilities, can compensate for the lack of information in the occluded areas by leveraging multi-view global information, achieving more accurate detail restoration. Figure 4 As shown, in wide-baseline scenes, this method, combined with depth estimation, effectively improves the accuracy of viewpoint alignment and detail reconstruction: the edges of objects in its super-resolution results are clearer, and the error is concentrated only in local areas, avoiding the blurring and distortion problems caused by insufficient depth estimation in traditional methods. Furthermore, from the perspective of local details, in the visual contrast of plant scenes, the super-resolution results of this method show significantly better restoration of leaf textures and object contours than methods such as DistgASR and ELFR, further confirming its comprehensive advantages in information modeling and scene adaptability.

[0119] Table 1 arrive Quantitative Analysis of Different Algorithms in Light Field Angle Super-Resolution Task

[0120]

Claims

1. A depth geometry prior-guided Mamba optical field angle super-resolution method, characterized in that, Includes the following steps: S1, Depth Estimation: A PSV is constructed based on sparse input viewpoint images, and a disparity map corresponding to the input viewpoint images is obtained through a depth estimation module, which serves as a priori geometric structure of the scene. The depth estimation module adopts a hierarchical, progressive convolutional architecture; S2, cross-modal adaptive fusion: The input viewpoint image and disparity map are encoded in a dual-branch parallel manner through the feature fusion module, and the adaptive weights of the encoded features are calculated to output a fused feature representation with depth perception characteristics. The feature fusion module includes an image convolution module, a depth convolution module, a channel gating module, and a spatial gating module; S3, Joint Feature Extraction of Empty Angles: A feature extraction model is used to perform multi-level deep feature extraction on the fused feature representation, and output the joint features of space and angle. The feature extraction model adopts a progressive enhancement architecture of cascaded grouping and hierarchical feature groups; S4, angle upsampling: The upsampling module performs angular upsampling on the joint features of the empty angle to output a high-resolution light field image. The upsampling module adopts a PixelShuffle 2D structure; S5, Joint Optimization: End-to-end optimization of the depth estimation model and the feature extraction model is performed using a joint loss function; The joint loss function includes image reconstruction loss and depth consistency constraint.

2. The Mamba optical field angle super-resolution method guided by depth geometry priors according to claim 1, characterized in that, In S1, the process of obtaining the disparity map corresponding to the input viewpoint image is as follows: S11, PSV Construction: Constructing the PSV at the input viewpoint based on the sparse input viewpoint image: First, the rearrangement operation: rearrange the input viewpoint image. Reconstructing sub-aperture images from sub-aperture image array format ,in For the set of real numbers, For batch size, The input angular dimension is h, and the width and height of a single sub-aperture image are h and w, respectively. Then, a reverse deformation construction mechanism is used to construct the sub-aperture image. Corresponding PSV: Based on preset Discrete disparity planes By reprojecting sub-aperture images , will input Arbitrary sampling positions and the synthesized target positions are simultaneously encoded into the PSV to generate a set of deformed images corresponding to each disparity plane. Sub-aperture image The corresponding PSV; among which, The number of discrete disparity planes. For a single disparity plane, The images obtained after reprojection and distortion of each parallax plane; S12, Disparity Regression: Based on Sub-Aperture Images The corresponding PSV, combined with multi-view stereo vision constraints, estimates the disparity map of the input viewpoint. ; The parallax regression module performs feature learning and information aggregation from three dimensions: intra-view, inter-depth, and spatial context. The disparity regression module adopts a hierarchical and progressive convolutional architecture, which includes a plane-by-plane convolution module, a cross-plane convolution module, and a disparity prediction module. First, the plane-wise convolution module reconstructs the PSV obtained in S11 into The two-dimensional feature mapping is used to learn view-invariant feature representations by independently fusing multi-view information on each depth plane through four cascaded convolutional layers. Then, the cross-plane convolutional module shifts the focus to consistency modeling of the depth dimension by flattening the depth dimension into a channel dimension and learning continuity constraints in the depth hypothesis space using the equivalent form of three-dimensional convolution. Finally, the disparity prediction module concatenates all features from each disparity plane and predicts disparity values ​​through consecutive convolutional layers, outputting a disparity map. .

3. The Mamba optical field angle super-resolution method guided by depth geometry priors according to claim 1, characterized in that, In S2, the cross-modal adaptive fusion process is as follows: S21, input viewpoint image With disparity map As input, the apparent texture features are obtained through dual-branch parallel encoding using an image convolution module and a depth convolution module. with depth features ; The image convolution module is used to preserve the texture details and color fidelity of the image, for the input viewpoint image. Two-stage convolutional coding is performed; the image convolution module uses two convolutional layers with a kernel size of 3×3 and a stride of 1. ReLU activation is used between the convolutional layers for non-linear activation, outputting apparent texture features. ; The deep convolution module is used to extract the geometric structure and spatial relationships of the scene, and to process the disparity map. Three-level convolutional encoding is performed; the depthwise convolutional module uses three convolutional layers with a kernel size of 3×3 and a stride of 1. ReLU activation is used between convolutional layers for non-linear activation, outputting depth features. ; S22, Calculate the disparity map mean and standard deviation The normalized disparity map is obtained by normalizing the data. The Sobel operator is used to compute the normalized disparity map. The horizontal and vertical gradients are used to obtain the depth geometric structure features. ; For deep geometric features Perform edge enhancement: enhance apparent texture features Depth features Normalized disparity map and depth geometric features The features are concatenated along the channel dimension and fed into a feature fusion convolutional network to output geometrically enhanced features with reliable geometric priors. ; The Sobel operator includes and ; and The convolution kernels are all 3×3; ， ； The feature fusion convolutional network includes two convolutional layers, each with a kernel size of 3×3 and a stride of 1. The ReLU activation function is used between the convolutional layers. S23, Appearance texture features based on gating attention mechanism and depth features Perform adaptive fusion to output gated weighted geometric enhancement features and fusion weights; First, global mean pooling is used to enhance the geometric features. Pooling as a geometric enhancement feature Then, after passing through the channel gating module, the channel attention weights are output. ; The channel gating module includes cascaded channel compression convolution and channel expansion convolution; the activation function of the channel compression convolution is the ReLU activation function, and the kernel size is 1×1; the activation function of the channel expansion convolution is the Sigmoid activation function, and the kernel size is 1×1. Then, pooling geometric enhancement features Normalized disparity map and depth geometric features The data is concatenated along the channel dimension, input to the spatial gating module, and output as spatial attention weights. ; The spatial gating module includes two cascaded convolutional layers; the first convolutional layer has a kernel size of 3×3 and uses the ReLU activation function; the second convolutional layer has a kernel size of 1×1 and uses the Sigmoid activation function. The pooled geometric augmentation features and channel attention weights after channel gating are combined. After multiplication, it is then multiplied by the spatial attention weight. splicing in the channel dimension; The concatenated result is input into the convolutional layer to obtain the fusion weights. ; The convolutional layer has a kernel size of 1×1 and uses the Sigmoid activation function. S24, pooling geometry enhancement features With fusion weight The enhancement is obtained by multiplication; then residual fusion is applied to the input viewpoint image. The summation outputs a fused feature representation with depth-aware characteristics. .

4. The Mamba optical field angle super-resolution method guided by depth geometry priors according to claim 1, characterized in that, In S3, the process of extracting the joint feature of empty angles is as follows: The progressive enhancement architecture includes four sets of cascaded feature modules, each set of cascaded modules includes four sets of hierarchical feature modules, and each hierarchical feature module is a CNN-Mamba multi-branch feature extraction block; The CNN-Mamba multi-branch feature extraction block includes an angular dimension geometric perception aggregation submodule, a depth-constrained EPI structure consistency extraction submodule, a spatial high-frequency detail preservation submodule, a geometric edge response submodule, and a geometrically enhanced sequence state space global modeling submodule. S31, fusing feature representations through geometrically guided CNN-Mamba multi-branch feature extraction blocks. Parallel extraction of multi-dimensional spatial features: S311, the angle dimension geometric perception aggregation submodule aggregates the correlation between different viewpoints of the light field under the prior constraints of geometric structure; The angle-dimensional geometry-aware aggregation submodule sequentially includes a non-filled convolutional layer and a convolutional layer; the kernel of the non-filled convolutional layer is α×α, and the stride is α; the kernel size of the convolutional layer is... The number of output channels is 64× ; First, padding-free convolutions are used to represent the fused features. Angular dimension structured downsampling is performed to obtain feature maps, thereby achieving the aggregation of geometric consistency information within the local viewpoint neighborhood; Then, the features are mapped to a high-dimensional angle space through a convolutional layer to obtain the channel-dimensional angle encoding information; Finally, pixel rearrangement operations are used to rearrange the angle encoding information from the channel dimension to the spatial dimension, achieving synergistic integration of geometrically perceived angle features and spatial dimensions, and outputting angle-enhanced features. ; S312, guided by depth geometry priors, the depth-constrained EPI structure consistency extraction submodule extracts the structural consistency features of the light field in the horizontal EPI and vertical EPI directions respectively. The depth-constrained EPI structure consistency extraction submodule includes two parallel EPI branches: a horizontal EPI branch and a vertical EPI branch. The horizontal EPI branch first passes through a convolution kernel with a size of 1× Step size is Fill as Convolutional layers for fused feature representation A sliding encoding along the horizontal viewpoint direction is performed to capture the continuity of the horizontal epipolar structure under depth prior constraints; then, after dimensionality upscaling via 1×1 convolution, one-dimensional pixel rearrangement is used to output the high-dimensional EPI features along the horizontal viewpoint direction. ; The vertical EPI branch will fuse feature representations. After performing spatial dimension swapping and executing the same processing flow as the horizontal EPI branch, the original dimensions are restored, and the EPI structural features in the vertical view direction are extracted. ; S313, the spatial high-frequency detail preservation submodule, employs a double-dilated convolutional structure. By expanding the receptive field, it captures neighborhood spatial context information constrained by geometric structure. Combining the ReLU activation function with a 3×3 spatial feature enhancement convolution, it selectively enhances geometric boundaries and high-frequency texture details, outputting spatial detail enhancement features. ; The porosity of the double-hole convolutional structure is... Fill with ; S314, the geometric edge response submodule includes three types of high-frequency operators with fixed convolution kernels. , and First, the three types of high-frequency operators respectively represent the fusion features. Each channel is independently subjected to grouped convolution to obtain three types of high-frequency features; these three types of high-frequency features include horizontal gradient features, vertical gradient features, and second-order high-frequency texture features; then, the three types of high-frequency features are concatenated along the channel dimension and a convolution kernel of size [size missing] is applied. Output channels are Convolution is used to compress features and obtain high-frequency features. ; ， ， ； S315, the geometry-enhanced sequence state space global modeling submodule adopts the Mamba architecture for fused feature representation. Long-program sequence dependency modeling is performed, treating the features of each viewpoint of the light field as sequence elements. Leveraging the global receptive field advantage of the state-space model, global angular correlations across viewpoints are captured under geometric prior constraints, outputting globally consistent features. ; The Mamba architecture is used to represent the fusion features. The specific process of performing long-program column dependency modeling is as follows: First, the fusion feature representation Perform multi-directional rearrangement: merge feature representations According to the preset space-angle division rules, each direction is expanded into at least four direction sequences, forming a set of direction sequences. The first direction sequence is achieved by adjusting the input tensor according to... Spatial combination rearrangement And flattened to obtain; the second direction sequence is obtained by pressing the input tensor according to Rearranged as The sequence is obtained by flattening the sequence; the third direction sequence is obtained by reversing the first direction sequence; the fourth direction sequence is obtained by reversing the second direction sequence. in, For general sequence variables, This is used to indicate the sequence direction; u and v represent the number of horizontal and vertical input viewpoints, respectively. Then, sequence projection transformation: for the set of direction sequences Linear projections are performed in each direction to obtain the time step parameter sequence. State input sequence and output weight sequence The three sets of projection features; Next, selectively scan the state update: update the set of direction sequences. Time step parameter sequence State decay matrix State input sequence and output weight sequence The input is fed into the selective scan module, where a directional output sequence is generated using a recursive method based on the sequence state space of the decay kernel. ; The selective scan module achieves differentiable forward and backward propagation through CUDA core operations. Next, multi-directional feature merging: directional output sequences are then processed. Perform a weighted summation in each direction, and then use an inverse transformation to restore the weighted summation result to the three-dimensional feature tensor y corresponding to the input size; Finally, feature normalization: for the three-dimensional feature tensor Perform normalization processing to output globally consistent features. ; S32 enhances the angle feature output by S311. High-dimensional EPI features in the horizontal view direction output by S312 EPI structural features in the vertical viewing direction output by S312 S313 output spatial detail enhancement features High-frequency characteristics of S314 output and the global consistency characteristics of S315 output The features are concatenated along the channel dimension; then, a channel attention mechanism is used to weight the concatenated multi-branch fusion features channel by channel, dynamically adjusting the contribution ratio of local high-frequency details and global angular consistency; finally, a 1×1 convolution is used for channel compression to output the joint features of empty and angled regions. This is used for subsequent angle upsampling.

5. The Mamba optical field angle super-resolution method guided by depth geometry priors according to claim 1, characterized in that, In S4, the angle upsampling process includes: The upsampling module includes a first-layer channel-compressed convolution, a second-layer channel-expanded convolution, a third-layer PixelShuffle layer, and a fourth-layer channel-compressed convolution; the kernel size of the first-layer channel-compressed convolution is... Step size is The activation function is ReLU; the kernel size of the second-layer channel-expanded convolution is... The number of channels is 64× The rearrangement factor of the third PixelShuffle layer is: The fourth layer channel compression convolution has a kernel size of 3×3 and an inflation rate of 100%. Fill with ;in, To output the angle dimension; S41, using the upsampling module to perform joint features of the empty angle. Angular upsampling is performed to output a high-resolution light field image in macro image format. ; S42, high-resolution light field image Convert the image format to sub-aperture format. .

6. The Mamba optical field angle super-resolution method guided by depth geometry priors according to claim 1, characterized in that, In S5, the joint optimization process is as follows: Computational sub-aperture format image and the image of the viewpoint to be tested loss function between , obtain the loss value Among them, the loss function Including absolute value loss function EPI Consistency Loss Function ; The loss value Adam is used as a supervisory term for the depth estimation and feature extraction modules; it is selected as the optimizer to adjust the weights and biases of the depth estimation and feature extraction modules to minimize the loss value. ; First, the loss value is calculated using the backpropagation algorithm. For each weight and bias, the gradient; Then, the parameters are updated using gradient descent, the training data is input into the light field angle super-resolution network, and the predicted value is calculated through forward propagation; The light field angle super-resolution network is an overall network structure composed of a depth estimation module, a feature fusion module, a feature extraction model, and an upsampling module. Next, the parameters are updated through backpropagation, and the process is iterated until the light field angle super-resolution network converges. All data in the training set are trained once as one cycle. After training, the trained depth estimation module and feature extraction module are obtained. The loss function The calculation is as follows: ； in, This represents the absolute value loss between the predicted image and the image to be tested. For EPI consistency loss, and They are respectively and The proportion of the loss function.

7. An optical field angle super-resolution network utilizing any one of the optical field angle super-resolution methods according to claims 1 to 6, characterized in that, The light field angle super-resolution network sequentially includes a depth estimation module, a feature fusion module, a feature extraction model, and an upsampling module. The depth estimation module is used to obtain the disparity map corresponding to the input viewpoint image. The feature fusion module is used to encode and fuse the disparity map with the input viewpoint image to obtain a fused feature representation. The feature extraction model is used to perform multi-level depth feature extraction on the fused feature representation to obtain joint features of space and angle. The upsampling module is used to perform angle upsampling on the joint features of space and angle to obtain a high-resolution light field image.

8. An electronic device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images