A light field angular super-resolution reconstruction method for shielding perception condition diffusion

Abstract

A super-resolution reconstruction method for light field angles based on occlusion-aware conditional diffusion is disclosed, belonging to the field of light field computational imaging. Based on a low-angle resolution light field estimation disparity map, occlusion maps for each view are generated using disparity-guided distortion residuals based on photometric consistency. Features of the low-angle resolution light field and occlusion maps are extracted by an occlusion-aware conditional encoder and upsampled to the target angular resolution to obtain a conditional map. A view-level two-dimensional position encoding is then applied to construct a position-aware conditional signal. The noisy light field and conditional signal are input into a decoupled, empty-angle structured diffusion denoising network. Simultaneously, diffusion time steps are embedded and injected into each layer of the network. A Transformer block with cross-attention is used in the bottleneck layer of the diffusion denoising network to perform cross-attention modulation on intermediate features using conditional tokens. Noise is estimated and iterative denoising is performed to obtain the target angular resolution light field. This application improves the visual quality of light field viewpoint synthesis while maintaining geometric consistency.

Description

Technical Field

[0001] This invention relates to the fields of light field computational imaging, viewpoint synthesis and diffusion modeling, and specifically to a method for super-resolution reconstruction of light field angles under occlusion-sensing conditions. Background Technology

[0002] Light field cameras can record the intensity and direction of light in real-world scenes, providing additional angular information for the 3D representation of real-world objects. High angular resolution light field images contain rich 3D geometric information, a property highly sought after in many applications such as 3D reconstruction, depth estimation, refocusing, and virtual reality. Early light field imaging techniques used camera arrays to capture light field images, allowing for the acquisition of high angular resolution images by assembling a large number of cameras. However, this was expensive and bulky, making it inconvenient for practical or commercial light field image acquisition. In recent years, all-optical cameras have become more commonly used, making light field image acquisition more convenient. However, due to angle-spatial multiplexing on image sensors, angular resolution still falls short of satisfactory levels, inevitably leading to a significant degradation in spatial resolution.

[0003] To alleviate this predicament, improving the angular resolution of light field images has attracted widespread attention. Unlike spatial super-resolution of light field images, angular super-resolution aims to generate dense light fields from sparsely sampled light field data. Although the viewpoint sampling of sparse light fields is relatively limited, each viewpoint still carries information about the geometric structure and radiation characteristics of the scene. This information is highly correlated between viewpoints and can collaboratively recover a complete representation of the scene. For example, object information missing from one viewpoint due to occlusion can be compensated for from data from other viewpoints. Therefore, reconstructing dense viewpoint light fields based on sparsely sampled light fields is not only theoretically feasible but also has significant practical application implications. Early research was mostly based on the frequency domain characteristics or geometric constraints of light field signals, optimizing interpolation functions through explicit modeling. These methods can achieve good reconstruction results when there are a sufficient number of input viewpoints, but their performance is significantly limited when the number of input viewpoints is small. In recent years, deep learning, as a powerful tool extending from neural networks, has greatly promoted research on angular super-resolution of light fields. These methods typically utilize convolutional neural networks or generative models to learn the mapping relationships between viewpoints end-to-end from sparse input viewpoints, significantly improving the synthesis quality of dense viewpoints. However, the performance of these methods in real-world scenes is often limited, failing to accurately infer edge occlusion regions, resulting in blurred and artifact-laden views. Furthermore, they remain significantly limited when dealing with challenging scenes such as large parallax light fields and complex textures. These issues severely hinder the widespread adoption of light field technology. Therefore, developing technologies and systems capable of achieving high-quality light field viewpoint synthesis in complex scenes has become a core problem urgently needing to be solved.

[0004] Considering that the diffusion model consists of forward and backward processes, it can simulate the complex distribution of various modalities and achieve high-quality generation with realistic details. Furthermore, by introducing conditional signals, the diffusion model allows for accurate and flexible control of the generation process. Based on these advantages, the diffusion model shows great potential in generating dense light field viewpoints from sparse light field viewpoints. Therefore, the diffusion model is used for light field viewpoint synthesis, and the disparity map is used to estimate the occlusion map for explicit occlusion processing, thereby improving the reconstruction effect of dense light fields. Compared with existing convolutional network methods for light field angle super-resolution reconstruction, the network of this invention achieves more accurate and flexible conditional control through the diffusion model, reduces dependence on inaccurate information through generative methods, and improves reconstruction accuracy. Compared with the typical light field angle super-resolution reconstruction diffusion model paper LFdiff, the network of this invention enhances the perception of occluded regions, improves the geometric consistency of the reconstructed light field, and captures long-range dependencies through explicit occlusion control, a pre-trained encoder, and a further improved diffusion denoising network, thus improving the reconstruction effect.

[0005] The present invention aims to solve at least one of the following technical problems: geometric ambiguity and boundary artifacts caused by occlusion and parallax inconsistency during the synthesis of light fields from low-angle resolution light fields to high-angle resolution light fields; insufficient control over the spatial coupling structure and deep semantic consistency of four-dimensional light fields when the diffusion model only performs shallow stitching; and how to stably combine light field conditional encoding and diffusion denoising networks under training resource constraints. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a super-resolution reconstruction method for optical field angles using occlusion-aware conditional diffusion, belonging to the field of optical field computational imaging. Based on a low-angle resolution optical field estimation disparity map, occlusion maps for each view are generated using disparity-guided distortion residuals based on luminance consistency. Features of the low-angle resolution optical field and occlusion maps are extracted by an occlusion-aware conditional encoder and upsampled to the target angular resolution to obtain a conditional map. A view-level two-dimensional position encoding is then applied to construct a position-aware conditional signal. The noisy optical field and conditional signal are input into a decoupled, empty-angle structured diffusion denoising network. Simultaneously, diffusion time steps are embedded and injected into each layer of the network. A Transformer block with cross-attention is used in the bottleneck layer of the diffusion denoising network to perform cross-attention modulation on intermediate features using conditional tokens. Noise is estimated and iteratively denoised to obtain the target angular resolution optical field. This application explicitly senses the occlusion region, injects angular position information, and strengthens conditional control through cross-attention, which helps improve the visual quality of optical field viewpoint synthesis while maintaining geometric consistency.

[0007] A super-resolution reconstruction method for the angle of light field under occlusion-sensing conditions, as shown in Figure 1, includes the following steps:

[0008] Step S1, Obtain input data: The input data is a low-angle resolution light field;

[0009] Step S2: Obtain the occlusion map of the low-angle resolution light field based on the parallax and luminance consistency.

[0010] Step S3: Input the low-angle resolution light field and occlusion map into the occlusion-sensing conditional encoder to obtain the angle-sensing conditional signal.

[0011] Step S4: Input the low-angle resolution light field into the diffusion model for forward noise addition to obtain the noisy light field;

[0012] Step S5: The noisy light field and the angle-sensing condition signal at the current time step are concatenated in the channel dimension and then input into the diffusion model for reverse iterative denoising, outputting the light field with the target angle resolution. .

[0013] Furthermore, in step S1, the process of acquiring input data is as follows:

[0014] S11, Obtain the publicly available light field datasets HCI-new and UrbanLF (or other light field datasets), wherein all light fields in the datasets are stored in the form of sub-aperture image arrays;

[0015] S12, select a 5×5 sub-aperture image array in the center region of each light field scene as a high-angle resolution reference ground value for model training and testing evaluation; for the 5×5 sub-aperture image array, sample every other view along the horizontal and vertical directions to obtain a 3×3 low-angle resolution sub-aperture image array.

[0016] S13, extract the Y channel of each sub-aperture image in the 3×3 low-angle resolution sub-aperture image array as the input of the model, and denote the low-angle resolution light field as... Angular resolution is (here) The spatial resolution of each sub-aperture image is The reconstructed high-angle resolution light field of the target is denoted as... Angular resolution is (here) The spatial resolution of each sub-aperture image is ,like Figure 2 As shown.

[0017] Furthermore, in step S2, the process of obtaining the occlusion map is as follows:

[0018] S21, Center view based on low-angle resolution light field , Given a set of real numbers; a center view is obtained using a pre-trained monocular depth estimator. Normalized inverse depth ; inverse depth Rescale to the desired parallax range to obtain the parallax map of the center view. ;

[0019] S22, excluding the center view from the low-angle resolution light field. The remaining sub-aperture images (i.e., non-centered sparse input views) are based on the disparity map. Distort the image to the center view coordinate system to obtain the distorted image. ;

[0020] S23, Calculate the distorted image With center view The absolute residuals between them are normalized or linearly mapped to obtain a photometric consistency measure;

[0021] S24, perform a power-law transformation on the photometric consistency metric to generate occlusion maps M for each view in the low-angle resolution light field. The power-law transformation is used to balance occlusion sensitivity and noise robustness, such as... Figure 3 As shown.

[0022] Furthermore, in step S3, the occlusion-aware conditional encoder includes an initial feature extractor, a feature fusionist, and an EPIT angle super-resolution network;

[0023] The initial feature extractor includes a light field branch initial extractor and an occlusion branch initial extractor; the light field branch initial extractor and the occlusion branch initial extractor are independent of each other, the light field branch initial extractor extracts the initial features of the light field image, and the occlusion branch initial extractor extracts the initial features of the occlusion image;

[0024] The feature fusion unit fuses the initial features output by the initial feature extractor and adjusts them to a specific number of channels to obtain fused features; the specific number of channels is 64.

[0025] The EPIT angle super-resolution network includes an alternating empty-angle filter block, an upsampling module, and an encoder; the fused features are processed by the EPIT angle super-resolution network to obtain an angle-sensing condition signal.

[0026] Furthermore, in step S3, the process of acquiring the angle sensing condition signal is as follows:

[0027] S31 will reduce the low-angle resolution light field The occlusion map M and the occlusion-aware conditional encoder are sequentially input into the light field branch feature extractor. With occlusion branch feature extractor This yields the initial characteristics of the low-angular-resolution light field. Initial features of the occlusion map ; and Their shapes are exactly the same;

[0028] S32, initial features and After concatenation along the channel dimension, the data is input into a feature fusion processor for fusion to obtain the fused features. ;

[0029] S33 will integrate features The signal is input into the EPIT angle super-resolution network for angle super-resolution processing to obtain the angle sensing condition signal c.

[0030] Furthermore, in step 33, the angle super-resolution processing procedure is as follows:

[0031] First, the empty-angle alternating filter blocks fuse the features. Deep spatial-angle correlation learning is performed to obtain learned features.

[0032] Then, the upsampling module upsamples the angular resolution of the learned features to the target angular resolution, and adjusts the learned features into a single channel through convolution, which is then compared with the low angular resolution light field. The conditional graph is obtained by adding the bicubic upsampling results of the angle dimension;

[0033] Finally, the view-level two-dimensional sine-cosine position encoding is superimposed on the condition map to obtain the angle-aware condition signal c.

[0034] The two-dimensional sine-cosine position encoding is used for each sub-aperture image based on its coordinates in the angle grid. A fixed encoding vector, unaffected by spatial location, is generated. All pixels within the same view share this encoding, enabling the model to distinguish between different views while maintaining spatial consistency within the view.

[0035] Furthermore, in step S4, the diffusion model includes a forward phase and a reverse phase;

[0036] The forward phase: random Gaussian noise is gradually added to the low-angle resolution light field to obtain the intermediate state light field, which is called the noisy light field;

[0037] The reverse stage, also known as the reverse denoising stage, employs a diffusion denoising network for iterative reverse denoising to reconstruct the light field at the target angular resolution. The diffusion denoising network used is DistgU-ViT.

[0038] The noisy light field at the current time step and the angle-sensing condition signal are concatenated in the channel dimension and input into DistgU-ViT. The projected diffused time steps are injected into each level of DistgU-ViT, which is called time step embedding. DistgU-ViT outputs noise estimation and determines the noise intensity of the current time step according to a preset diffusion schedule (e.g., linear or cosine schedule). Through iterative denoising, the light field with the target angular resolution is gradually restored. This enables the synthesis of sparse angle views into dense angle views.

[0039] In traditional diffusion models, diffusion denoising networks are typically implemented using the Unet architecture. This method designs an improved Unet scheme adapted to 4D light field structures—DistgU-ViT. DistgU-ViT consists of DistgResBlock decoupled residual blocks. The core computational unit of DistgResBlock is DisentgBlock, which can decouple 4D light field information into multiple 2D subspaces for parallel / alternating modeling based on the light field features represented by MacPI. At the same time, Transformer blocks and cross attention are introduced in the bottleneck layer of DistgU-ViT (i.e., the layer with the highest degree of feature abstraction) to achieve explicit modulation of deep features by conditional information.

[0040] Furthermore, DistgU-ViT includes an initial convolution, a decoupled residual block, a bottleneck layer, and a lightweight conditional context convolutional network, as shown in Figure 5.

[0041] Furthermore, in step S5, the reverse denoising process is as follows:

[0042] S41, the noisy light field and angle-sensing condition signal c of the current time step are reconstructed from the sub-aperture image into macro-pixel form, then stitched in the channel dimension, input into DistgU-ViT, and after initial convolution, the channels are adjusted to 64 dimensions to obtain a 64-dimensional signal;

[0043] S42, the decoupling residual block DisgResBlock performs multi-scale feature extraction on the 64-dimensional signal; the decoupling residual block decouples the coupled optical field information to different domains, including the spatial domain, angular domain, horizontal polar plane domain and vertical polar plane domain; as shown in Figure 4;

[0044] S43, the bottleneck layer uses a Transformer block; the Transformer block includes self-attention, cross-attention, and feedforward networks; wherein, the query of cross-attention comes from the feature token of the current layer;

[0045] S44, the Conditional Context Convolutional Network downsamples the angle-aware conditional signal c in terms of spatial dimension and transforms it in terms of channel dimension to extract compact semantic conditional tokens; the conditional feature map is compressed by the Conditional Context Convolutional Network to obtain a conditional token sequence; the key and value of the cross attention are derived from the conditional token sequence.

[0046] Through the cross-attention mechanism, conditional information can explicitly modulate deep features at the most semantically abstract layer of the network, thereby guiding the denoising process to generate a high-angle resolution light field that conforms to the scene structure.

[0047] S45, based on the noise estimation and diffusion scheduling of DistgU-ViT output (i.e., the predefined noise intensity sequence during the forward noise addition process), subtract the predicted noise component from the noise-added optical field at the current time step to obtain the optical field estimate of the previous time step.

[0048] S46, repeat S41 to S45, iterating step by step from t=T to... Finally, the target angular resolution light field is obtained. T represents the total training time step. This is the diffusion time step.

[0049] In an optional implementation, the training process is divided into two stages: first, the occlusion-aware conditional encoder is trained or pre-trained with reconstruction or feature alignment as the main target; then, the network parameters of the occlusion-aware conditional encoder are fixed, and the diffusion denoising network is trained with the mean square error of the predicted noise as the main target.

[0050] In an optional implementation, the input is converted from RGB to luminance-chrominance space, the angle super-resolution diffusion reconstruction is performed only on the luminance channel, and the chrominance channel is interpolated and upsampled in the angle dimension and then synthesized with the reconstructed luminance channel for output.

[0051] Compared with existing technologies, the technical effects of this invention are as follows: Compared with schemes that rely solely on implicit network fitting without explicitly utilizing occlusion and angular geometric priors, this invention constructs an occlusion map through photometric consistency guided by parallax, which is beneficial for enhancing the perception of unreliable correspondences and occlusion boundaries in conditional branches; Compared with schemes that only stitch conditions at the network entry point without explicit interaction in deeper layers, introducing a transformer and cross-attention in the deepest layer of the diffusion denoising network and modulating features with conditional tokens is beneficial for global consistency and conditional controllability; The occlusion-aware conditional encoder adopts an alternating empty-angle structure and angle upsampling, which is connected to the decoupled diffusion backbone in the form of macropixels, which is beneficial for adapting to four-dimensional light field data structures; Experiments show that in angular super-resolution tasks ranging from 3×3 to 5×5, high peak signal-to-noise ratio and structural similarity indices can be obtained on synthetic and real datasets, and more reasonable visual results can be obtained at occlusion boundaries. Attached Figure Description

[0052] Figure 1 is a schematic diagram of the network structure of the present invention;

[0053] Among them, Figure 1(a) shows the backbone of the light field angle super-resolution diffusion model, and Figure 1(b) shows the specific structure of the occlusion perception condition encoder.

[0054] Figure 2 For 4D representation and visualization of light fields;

[0055] Figure 3 This is a detailed structural diagram of the occlusion image generator;

[0056] Figure 4 shows the decoupled residual block structure of the diffusion denoising network;

[0057] Figure 4(b) shows the decoupled residual block DistgResBlock, and Figure 4(a) shows the core computing unit DisentgBlock of DistgResBlock.

[0058] Figure 5 is a comparison of the architectures of DisgU-ViT and DisgUnet;

[0059] Figure 5(a) is the architecture diagram of DistgU-ViT; Figure 5(b) is the architecture diagram of DistgUnet.

[0060] Figure 6 Example of qualitative comparison and error plot for experiments;

[0061] in, Figure 6 (a) are ground truth images (GT) at different locations in two different images; Figure 6 (b) is a graph showing the experimental results of the LFdiff method; Figure 6 (c) is a diagram showing the experimental results of the present invention. Detailed Implementation

[0062] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0063] The embodiments described in this section are for illustrative purposes only and do not constitute a limitation on the scope of protection of this invention. Those skilled in the art can make equivalent substitutions and modifications without departing from the concept of this invention, and all such substitutions and modifications should be covered within the scope of protection of this invention. Parts not described in detail herein can be implemented using known techniques or conventional methods in the art.

[0064] Terminology Explanation:

[0065] Sub-aperture images (SAIs): A collection of two-dimensional images obtained by unfolding the light field according to the viewing angle.

[0066] Macropixels (MacPI): Two-dimensional image blocks formed by arranging pixels at the same spatial location under different viewpoints according to fixed rules, which makes it easier to represent four-dimensional light fields as tensors suitable for two-dimensional convolution and attention processing.

[0067] Epipolar Image (EPI): An image obtained by fixing one spatial dimension and varying the angular dimension, used to characterize the geometric consistency in a light field.

[0068] Low-angle-resolution light field: refers to a light field data in which the number of viewpoint samples is small and the angular direction distribution is sparse. For example, the angular resolution is only 3×3 or less (i.e., no more than 9 viewpoints in total). Such sparsely sampled input light field is called a low-angle-resolution light field. The task of this application is to reconstruct a complete light field with dense viewpoints and higher angular resolution from such limited and sparse viewpoint information.

[0069] Center view: refers to the view located at the geometric center in a sparse input light field view grid. For example, for a 3×3 input view grid, the center viewpoint is the view in the 2nd row and 2nd column.

[0070] Target angular resolution: refers to the number of samples of the dense light field expected to be obtained in the angular direction after light field reconstruction. For example, 5×5 means there are 5 viewpoints in the horizontal direction and 5 in the vertical direction, for a total of 25 views.

[0071] Diffusion time step (t): In the diffusion model, the continuous noise addition and denoising process is discretized into a finite number of steps, each step corresponding to an integer index t. Here, t=0 represents the original noise-free data, and t=T represents pure random noise; the forward process gradually adds noise, while the reverse process gradually denoises from t=T until t=0.

[0072] Luminosity-chromaticity space: A color model that decomposes an image signal into a luminance component representing brightness and a chromaticity component representing color. The luminance component carries the texture details and geometric structure of the image, while the chromaticity component carries hue and saturation information.

[0073] Disparity map: A two-dimensional image representing the horizontal displacement between corresponding pixels in the left and right images in stereoscopic vision. The disparity value reflects scene depth information and is inversely proportional to the depth map.

[0074] Existing light field angle reconstruction methods still have significant limitations when dealing with challenging scenarios such as large parallax light fields and severe occlusion. Therefore, this application provides a super-resolution light field angle reconstruction method for occlusion-perceived condition diffusion to address the aforementioned problems and achieve high-quality light field viewpoint synthesis.

[0075] The existing light field angle reconstruction methods described in this invention still have significant limitations when dealing with challenging scenarios such as large parallax light fields and severe occlusion. Therefore, this application provides a super-resolution light field angle reconstruction method for occlusion-perceived condition diffusion to solve the aforementioned problems and achieve high-quality light field viewpoint synthesis.

[0076] Before introducing the network structure of this invention, a brief description of the imaging model of the light field camera will be given. For example... Figure 2 As shown, the imaging model of the light field camera is based on a two-plane parameterized representation of the light field. The light field is represented by a 4D tensor, where light rays pass through and intersect at the angular plane. and spatial plane Microlens image arrays and sub-aperture image arrays are two typical examples of 4D light field visualization. The image shown in the center is a microlens image array. Magnifying it reveals a hexagonal grid reflecting the distribution of the microlens array. Each hexagonal grid, after removing redundancy, is called a macropixel. Extracting pixels at the same position from each macropixel and combining them forms a sub-aperture image array, such as... Figure 2 The image on the far right is shown. Let... Indicates a having Each spatial dimension is The light field of the sampled angle images (SAIs), then Indicates the first SAI .

[0077] Example

[0078] As shown in Figure 1, the overall process of the method of the present invention includes steps S1 to S4. Figure 1(a) shows the backbone of the optical field angle super-resolution diffusion model, and Figure 1(b) shows the specific structure of the occlusion-aware conditional encoder therein. The following describes each step in turn.

[0079] S1: Obtain input data

[0080] This step acquires a low-angular-resolution light field sub-aperture image array as input. The input data can be obtained directly from the sparse light field acquired by an all-optical camera, or by downsampling the high-angular-resolution light field in terms of angular dimension. The Y channel (luminance component) of each sub-aperture image is extracted as the actual input to the model, and this low-angular-resolution light field is denoted as... Its angular resolution is (In this embodiment) The spatial dimensions of each sub-aperture image are The reconstructed high-angle resolution light field of the target is denoted as... Its angular resolution is (here) The spatial resolution is the same as the input.

[0081] S2: Occlusion map estimation based on disparity and luminosity consistency

[0082] To reduce reliance on accurate semantic segmentation or ground truth depth, this step estimates a disparity map based on the central view of a low-angular-resolution light field. It then uses this disparity map to warp and align surrounding views to the central view, generating an occlusion map through photometric consistency residuals. This step is essentially equivalent to an occlusion map generator. Based on the occlusion prior that aligns the estimated disparity output with the viewing angle, the process is as follows: Figure 3 As shown. The specific implementation is as follows:

[0083] S21: Parallax plot of the estimated central view of the light field

[0084] Take input light field Central view The normalized inverse depth is estimated using a pre-trained monocular depth estimator (such as MiDaS). Then Rescale to the desired parallax range to obtain the parallax map of the center view. .

[0085] S22: Distort the coordinate system of the surrounding view to the center view.

[0086] For the kth surrounding view Based on Perform a warping operation, mapping it to the center view coordinate system to obtain a warped image. The process is represented as:

[0087]

[0088] in This indicates that the k-th view will be... Projected onto the center view The transformation operation.

[0089] S23: Calculate photometric consistency residuals

[0090] Hypothetical disparity map If it is accurate, then the projected view It should be the center view of the unobstructed area. They have the same value. Therefore, we use and The absolute residuals between them are used to measure photometric consistency.

[0091]

[0092] By normalizing or linearly scaling the residuals to an interval such as [0,1], a photometric consistency measure is obtained.

[0093] S24: Generate occlusion map

[0094] Performing a power transformation on the normalized residuals yields the occlusion map of the k-th view, which is essentially a weighted map representing pixel-level occlusion sensitivity and cross-viewpoint correspondence confidence.

[0095]

[0096] in The residual after mapping, It is a positive scalar used to control the sharpness of the response in the high residual region; in this embodiment, it is taken as... A compromise is achieved between occlusion sensitivity and noise robustness. Areas with relatively high median values ​​indicate significant differences from the central view after distortion, corresponding to occlusion, large parallax errors, or unreliable correspondences; relatively low median values ​​indicate consistent luminosity and reliable correspondences. Occlusion maps for the above views are shown. and This will be used as input for subsequent steps.

[0097] S3: Construct angle-aware conditional signals

[0098] The occlusion-aware conditional encoder consists of three parts: two initial feature extractors, a feature fusion unit, and an EPIT angular super-resolution network. This step converts the low-angular resolution light field... With occlusion map The input occlusion-aware conditional encoder, through initial feature extraction, feature fusion, and angle super-resolution, yields upsampled features corresponding to the target angular resolution. These features are then convolved to a single channel and compared with the low-angular-resolution light field. The angle-dimensional bicubic upsampling results are summed to obtain a condition map. This is then overlaid with view-level position encoding to obtain an angle-aware condition signal, which serves as explicit guidance for the diffusion backbone. The specific implementation is as follows:

[0099] S31: Extract initial features

[0100] Input low-angular resolution light field The occlusion map M calculated in step S2 is rearranged into a five-dimensional tensor form in the sub-aperture dimension. ,in For input angular resolution (in this embodiment) ), This is the single-view spatial dimension. Then, the light field branch feature extractor is input separately. With occlusion branch feature extractor The two branches do not share parameters and independently extract the low-angle resolution input light field. The initial features of the occlusion map M are obtained. Both branches use 3D convolutions with a kernel size of (1,3,3) for primary encoding. The light field branch additionally concatenates several 3D convolutions with a kernel size of (1,3,3) and LeakyReLU to form a shallow enhancement layer to improve the stability of texture representation and obtain the initial features of the input light field. Initial features of the occlusion map :

[0101]

[0102]

[0103] S32: Feature Fusion

[0104] Will and After channel-dimensional splicing, and via a feature fusion unit The feature tensor is obtained by fusion. :

[0105]

[0106] It consists of alternating 3D convolutions with LeakyReLU kernels of size (1,3,3) and 3D convolutions of size (3,3,3), and is jointly fused in both the angular and spatial dimensions to ensure contextual consistency between occluded and unoccluded regions. The number of feature channels after fusion is set to 64.

[0107] S33: Deep Void Angle Correlation Learning

[0108] The EPIT angle super-resolution network first performs deep spatial angle correlation learning, and then performs angle dimension upsampling. It first fuses the low-angle resolution light field with the occlusion map features. The input is used to perform further spatial-angular feature extraction on the EPIT angle super-resolution network. This feature extraction network consists of empty angle alternating filtering blocks from the EPIT method. Each block can be stably trained using residual connections. It is used for deep spatial-angular correlation learning, realizing coupled modeling of texture details in the same view and cross-view angle correlation, and enhancing robustness to disparity changes under sparse angle input conditions.

[0109] S34: Upsampling in the angular dimension

[0110] After performing deep spatial angle correlation learning, the features are fed into the upsampling module of the EPTI angle super-resolution network for angle-dimensional upsampling. In the angle dimension, let the input angle resolution be... The goal is At that time, the AngularUpsample module is configured to: first reshape the tensor into a two-dimensional angular plane in the angular dimension, then improve the angular resolution through angular-dimensional convolution and pixel shaving operations, and finally restore it to its original state. Individual aperture (in this embodiment) This yields the upsampled feature tensor. A convolutional layer at the end of the network adjusts the channels to a single channel, which is then compared with the low-angular resolution light field. The results of the bicubic upsampling at different angles are added together to obtain a conditional map used as the readout for the diffusion backbone, denoted as... .

[0111] S35: Overlay view-level position encoding

[0112] Angle-indexed 2D grid Sine-cosine periodic encoding (temperature coefficient can be 10000, etc.) is applied separately to achieve angular position distinguishability. The two-dimensional position encoding is then superimposed or stitched in the channel dimension and broadcast to the spatial region corresponding to each sub-aperture, resulting in a position map aligned with the spatial dimensions after macro-pixel expansion. Dimension is denoted as (16 dimensions in the embodiment). Specifically, for angular coordinates... The location encoding is defined as:

[0113]

[0114]

[0115] Therefore, position encoding provides the same positional information within the same view while distinguishing positions between different views. Finally, it connects to the conditional image output by the occlusion-aware conditional encoder. Together with the position code PE, a conditional signal c containing rich geometric information is formed:

[0116]

[0117] S4: Perform diffusion denoising and reconstruction

[0118] This step employs the Discrete-Time Step Denoising Diffusion Probability Model (DDPM) paradigm. It estimates noise and iteratively denoises using the DistgU-ViT diffusion denoising network, which decouples the empty angle information, outputting the target angular resolution light field. The specific implementation is as follows:

[0119] S41: Formalization of the diffusion model

[0120] The diffusion model employs the discrete-time step denoising diffusion probabilistic model (DDPM) paradigm. This model consists of a forward noise addition process and a backward denoising process: the forward process progressively injects Gaussian noise into the real light field samples, making its distribution approximate an isotropic Gaussian distribution; the backward process learns a parameterized conditional distribution to progressively recover the target angular resolution light field that satisfies the conditional constraints from the noisy samples. Unlike one-time regression reconstruction, the diffusion process, through multi-step iterative updates, exhibits better generation stability and detail fidelity in complex occlusion and high parallax scenes.

[0121] Noise is added as follows: Let the brightness component of the real target light field be... The total diffusion step size is The noise scheduling sequence is ,in ,definition , A forward-noisy Markov chain can be written as: Equivalent land, It can be obtained from the closed expression: Therefore, it can be seen that at any time step The noisy state can be determined by The noise is directly sampled from standard Gaussian noise, facilitating supervision during the training phase with random sampling time steps. The inverse denoising process learns the conditional distribution. ,in The angle sensing condition signal obtained in step S3.

[0122] In this embodiment, the parameterized diffusion denoising network is DistgUViT, which outputs noise estimation. .

[0123] S42: Network Input Preparation

[0124] Noisy light field and conditional signals The sub-aperture image is rearranged into macro-pixel format, stitched together along the channel dimension, and then input into the DistgU-ViT diffusion denoising network. In this embodiment, RGB is converted to the YCbCr color space, diffusion reconstruction is performed only on the Y channel, and the Cb and Cr channels are upsampled to the target angular resolution using bicubic interpolation in the angular dimension. This upsampling is then combined with the diffusion-reconstructed Y channel to synthesize the color result. For training sampling, a 5×5 sub-aperture in the center region of each scene in the training dataset is selected as the supervision region, consistent with the device or dataset annotations.

[0125] The DistgU-ViT diffusion denoising network de-noises a single-channel (or single-component) noisy optical field at the input. With conditional signals The sub-aperture image is reconstructed into macropixel form, stitched along the channel dimension, and then mapped to a 64-dimensional feature channel via a dilated or macropixel-aligned convolution before being input into DistgU-ViT. Simultaneously, the diffusion time step is... The embedding vector is generated by sinusoidal position encoding and two-layer linear mapping projection. This embedding vector will be injected into each subsequent DisgResBlock in an adaptive normalization or addition manner to modulate the intermediate features.

[0126] S43: Decoupling empty angle feature extraction

[0127] The DistgU-ViT diffusion denoising network uses DistgResBlock as its backbone, with DisentgBlock as its core computational unit. The structure of DisentgBlock is shown in Figure 4(a). For the light field features represented by MacPI, the 4D light field information is decoupled into multiple 2D subspaces for parallel modeling, including:

[0128] Spatial branch SpaConv: Two layers of 3×3 dilated convolution (dilation rate and filling angle resolution);

[0129] AngConv (Angle Branch): Angular domain convolution (kernel / stride of angRes) + 1×1 convolution + PixelShuffle;

[0130] EPI branch EPIConv: Convolution along the EPI direction + 1×1 convolution + one-dimensional PixelShuffle, extracting horizontal EPI and vertical EPI features respectively;

[0131] The fusion layer (fuse) concatenates the above branch features, fuses them through 1×1 convolution and 3×3 dilated convolution, and adds them to the input through residual / shortcut mapping.

[0132] This decoupling design captures domain-specific structures in interwoven four-dimensional light fields, enabling the explicit extraction of complementary "texture-geometry-angle" features at a lower computational cost, thus alleviating the problem of insufficient representation of complex light field structures by a single convolutional kernel.

[0133] The DistgResBlock structure is shown in Figure 4(b). It consists of a two-stage convolutional transform, time-step modulation, and residual connections. Its computational flow can be represented as follows:

[0134]

[0135] in , This indicates the main branch transformation that includes normalization, activation, and DisentgBlock. For time-embedded linear projection, This is a shortcut mapping. Specifically, The calculation order is 32 groups of GroupNorm normalization, Swish activation function, DisentgBlock, The computation order is: 32-group GroupNorm normalization, Swish activation function, Dropout random deactivation, and DisentgBlock. Dropout is used to suppress overfitting and improve training stability. In layers with attention configurations, self-attention submodules can be added to further enhance long-range dependencies. This structure enables the network to maintain consistent feature modulation capabilities across different diffusion steps, while stabilizing training through residual paths. In this embodiment, self-attention is enabled at the layer after the second downsampling of DistgU-ViT (i.e., the third layer).

[0136] S44: Bottleneck Layer Transformer Block and Cross Attention

[0137] As shown in Figure 5, unlike the LFdiff scheme which only stacks decoupled residual blocks at the bottom, this embodiment replaces the two DistgResBlocks of the original DistgUnet network in LFdiff with two cascaded Transformer blocks in the bottleneck layer (i.e., the layer with the highest degree of feature abstraction, i.e., the deepest layer of the network) of the diffusion denoising network DistgU-ViT.

[0138] Each block includes:

[0139] Tokenization: transforming feature maps Flattened ;

[0140] Layer normalization + multi-head self-attention (modeling global structure);

[0141] Layer normalization + multi-head cross attention (Query comes from the current feature token, Key / Value comes from the condition token);

[0142] Layer normalization + feedforward network (FFN);

[0143] Time-embedded modulation (scale / bias modulation is applied to the output).

[0144] Its core form of cross-attention is:

[0145]

[0146]

[0147]

[0148] in For condition token pairs, the key and value come from the sequence of condition tokens. This design allows conditional information to directly participate in feature updates at the most semantically abstract layer, which is more conducive to global geometric consistency control compared to shallow concatenation.

[0149] The conditional token is obtained from a lightweight conditional context convolutional network (CCNN). This lightweight CCNN, as a submodule of the diffusion denoising network, consists of several two-dimensional convolutions and activation functions. It downsamples the angle-aware conditional signal c in spatial dimension and transforms it in channel dimension to extract a compact semantic conditional token. This process is illustrated in the DistgU-ViT architecture in Figure 5. The conditional signal c is mapped by this lightweight CCNN to intermediate features with the same number of bottleneck feature channels. Then flatten it into a sequence of condition tokens:

[0150]

[0151]

[0152] in The number of condition tokens. Compared to concatenating conditions only at the downsampling entry and relying on subsequent convolutions for implicit fusion, this design explicitly modulates features with condition tokens at the bottleneck layer with the richest semantics and the largest receptive field. This is beneficial for adjusting deep representations based on the occlusion-aware condition map, improving geometric consistency and controllability.

[0153] S45: Noise Estimation and Iterative Denoising

[0154] Diffusion denoising network estimates time step Noise to be removed :

[0155]

[0156] Then, the mean square error of the actual noise injected during training at a given time step is calculated:

[0157]

[0158] When reasoning from Departure, according to Stepwise sampling yields the final reconstruction result. Under common parameterization, a one-step reverse update can be written as: ,in , Values ​​can be retrieved according to preset schedules.

[0159] Training strategy

[0160] This embodiment employs a two-stage training approach:

[0161] The first stage trains the occlusion-aware conditional encoder (the network in step S3) with reconstruction or feature-related loss to stabilize the occlusion fusion and angle upsampling output.

[0162] In the second stage, the parameters of the occlusion-aware conditional encoder are fixed, and only the diffusion denoising network (DistgU-ViT in step 4) is trained. The main loss is the noise prediction L2 loss, in order to reduce the optimization difficulty and avoid instability caused by the simultaneous drastic changes in conditional encoding and diffusion.

[0163] Specific training parameters: Total training time step Noise Dispatch from linearly increase to The diffusion model was trained using the AdamW optimizer with a batch size of 16 and an initial learning rate of 100%. It is scheduled by the cosine annealing scheduler.

[0164] Experimental results

[0165] To verify the performance of the network structure of this invention, specific experiments are conducted below to demonstrate the performance of the super-resolution network of this invention from multiple aspects, including qualitative and quantitative methods.

[0166] In a set of experiments, the task was to upgrade the angular resolution from 3×3 to 5×5. This invention uses two public datasets, HCI-new and UrbanLF, for training and to evaluate the effectiveness of the proposed method. Specifically, 16 scenes were selected from HCI-new and 170 scenes from UrbanLF. All light fields in these datasets have a 9×9 angular resolution. In the experiments, 5×5 views of the center of each scene were selected for training and testing. During training, each LF was cropped into a 5×5×32×32 patch. The network of this invention was evaluated on the synthetic datasets HCI-new and HCI-old and the real datasets STFGantry, EPFL, and Inria Lytro. For quantitative evaluation of network performance, two metrics, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), were used to calculate the score of the Y-channel super-resolution image. The super-resolution score for each scene was obtained by averaging the scores of all sub-aperture images (after super-resolution) within that scene. The super-resolution score for each dataset was obtained by averaging the scores of all scenes within that dataset. The quantitative indicators are shown in Table 1. This method has achieved excellent performance in both PSNR and SSIM indicators.

[0167] Table 1. Quantitative results of the network in this invention (3×3→5×5)

[0168]

[0169] Qualitatively, LFdiff, also a diffusion method, was selected for comparison. LFdiff synthesizes light fields from a single view to a 5×5 light field, and then re-infers the results to adapt to the angular super-resolution task of light fields from 3×3 to 5×5. The comparison results are as follows: Figure 6 As shown in the magnified local details, compared with the LFdiff comparison method, which does not fully utilize occlusion priors or lacks deep conditional interactions, this embodiment exhibits a more reasonable error distribution at occlusion boundaries. The errors are mainly concentrated near the boundaries and have minimal amplitude, effectively avoiding the problem of error diffusion to flat areas common in traditional methods. In complex edge regions (such as slender structures, high-texture abrupt changes, and weak-texture edges), this method achieves minimal reconstruction errors, significantly improving edge sharpness and sub-pixel-level positioning accuracy. Simultaneously, this embodiment demonstrates excellent geometric consistency, accurately preserving the parallax relationship between different viewpoints without significant structural distortion or parallax breaks. The overall reconstruction result exhibits strong visual naturalness and continuity, fully demonstrating the comprehensive advantages of this method in edge preservation, detail recovery, and cross-viewpoint geometric constraints. The above numerical values ​​are only used to illustrate the technical effects of this application and do not constitute a limitation on the scope of claims. Actual results vary depending on data partitioning, training hyperparticipant hardware, and other factors.

[0170] Extended Implementation

[0171] In another embodiment, a reflection estimation module and a non-Lambertian surface correction module can be added to the framework of this application to improve the viewpoint synthesis fidelity in complex material scenes such as specular surfaces and specular highlights. This extension can be combined with the above embodiments or as a divisional application; circuit and loss details are not elaborated here. Furthermore, the network of this invention can not only achieve optical field angular super-resolution from 3×3 to 5×5, but also achieve angular super-resolution reconstruction from 3×3 to higher magnifications such as 7×7 and 9×9 by modifying the target angular resolution of the model and simultaneously modifying the model's training data for retraining.

[0172] In summary, this application addresses the problems of unstable reconstruction of low-angle resolution light fields in occluded regions, insufficient deep conditional control, and limited training resources by proposing an integrated reconstruction framework of "occlusion-aware conditional encoding + view-level positional encoding + empty angle decoupling diffusion backbone + bottleneck cross-attention." This scheme explicitly constructs occlusion priors at the front end using parallax-guided photometric consistency. In the conditional encoding stage, it strengthens geometrically relevant features through dual-branch fusion and alternating empty angle learning. In the diffusion denoising stage, it maintains the multi-subspace structure modeling capability of the light field using DisentgBlock / DistgResBlock, and achieves precise modulation at the deep semantic level through TransformerMiddleBlock and conditional tokens. Experimental results show that this method can simultaneously improve geometric consistency and visual fidelity in super-resolution tasks with angles ranging from 3×3 to 5×5, and has better reconstruction results for complex occlusion boundaries and textured regions, demonstrating the method's innovation, effectiveness, and engineering feasibility.

Claims

1. A method for super-resolution reconstruction of the angle of a light field under occlusion-sensing conditions, characterized in that, Includes the following steps: Step S1, Obtain input data: The input data is a low-angle resolution light field; Step S2: Obtain the occlusion map of the low-angle resolution light field based on the parallax and luminance consistency. Step S3: Input the low-angle resolution light field and occlusion map into the occlusion-sensing conditional encoder to obtain the angle-sensing conditional signal. Step S4: Input the low-angle resolution light field into the diffusion model for forward noise addition to obtain the noisy light field; Step S5: The noisy light field and the angle sensing condition signal of the current time step are spliced ​​together in the channel dimension and then input into the diffusion model for reverse iterative denoising, and the light field with the target angle resolution is output.

2. The method for super-resolution reconstruction of the light field angle under occlusion-sensing condition diffusion according to claim 1, characterized in that, In step S1, the process of acquiring input data is as follows: S11, Obtain the publicly available light field datasets HCI-new and UrbanLF, where all light fields in the datasets are stored in the form of sub-aperture image arrays; S12, select a 5×5 sub-aperture image array in the center region of each light field scene as a high-angle resolution reference ground value for model training and testing evaluation; for the 5×5 sub-aperture image array, sample every other view along the horizontal and vertical directions to obtain a 3×3 low-angle resolution sub-aperture image array. S13, extract the Y channel of each sub-aperture image in the 3×3 low-angle resolution sub-aperture image array as the input of the model, and denote the low-angle resolution light field as... Angular resolution is The spatial resolution of each sub-aperture image is ; The reconstructed high-angle resolution light field of the target is denoted as Angular resolution is The spatial resolution of each sub-aperture image is .

3. The method for super-resolution reconstruction of the light field angle under occlusion-sensing condition diffusion according to claim 1, characterized in that, In step S2, the process of obtaining the occlusion map is as follows: S21, Center view based on low-angle resolution light field , Given a set of real numbers; a center view is obtained using a pre-trained monocular depth estimator. Normalized inverse depth ; inverse depth Rescale to the desired parallax range to obtain the parallax map of the center view. ; S22, excluding the center view from the low-angle resolution light field. The images of the remaining sub-apertures are based on the parallax diagram. Distort the image to the center view coordinate system to obtain the distorted image. ; S23, Calculate the distorted image With center view The absolute residuals between them are normalized or linearly mapped to obtain a photometric consistency measure; S24, perform a power transformation on the photometric consistency metric to generate occlusion maps M for each view in the low-angle resolution light field.

4. The method for super-resolution reconstruction of the light field angle under occlusion-sensing condition diffusion according to claim 1, characterized in that, In step S3, the occlusion-aware conditional encoder includes an initial feature extractor, a feature fusionist, and an EPIT angle super-resolution network; The initial feature extractor includes a light field branch initial extractor and an occlusion branch initial extractor; the light field branch initial extractor and the occlusion branch initial extractor are independent of each other, the light field branch initial extractor extracts the initial features of the light field image, and the occlusion branch initial extractor extracts the initial features of the occlusion image; The feature fusion unit fuses the initial features output by the initial feature extractor and adjusts them to a specific number of channels to obtain fused features; the specific number of channels is 64. The EPIT angle super-resolution network includes an alternating empty-angle filter block, an upsampling module, and an encoder; the fused features are processed by the EPIT angle super-resolution network to obtain an angle-sensing condition signal.

5. The method for super-resolution reconstruction of the light field angle under occlusion-sensing condition diffusion according to claim 1, characterized in that, In step S3, the process of acquiring the angle sensing condition signal is as follows: S31 will reduce the low-angle resolution light field The occlusion map M and the occlusion-aware conditional encoder are sequentially input into the light field branch feature extractor. With occlusion branch feature extractor This yields the initial characteristics of the low-angular-resolution light field. Initial features of the occlusion map ; and Their shapes are exactly the same; S32, initial features and After concatenation along the channel dimension, the data is input into a feature fusion processor for fusion to obtain the fused features. ; S33 will integrate features The signal is input into the EPIT angle super-resolution network for angle super-resolution processing to obtain the angle sensing condition signal c.

6. The method for super-resolution reconstruction of the light field angle under occlusion-sensing conditional diffusion according to claim 5, characterized in that, In step 33, the angle super-resolution processing procedure is as follows: First, the empty-angle alternating filter blocks fuse the features. Deep spatial-angle correlation learning is performed to obtain learned features. Then, the upsampling module upsamples the angular resolution of the learned features to the target angular resolution, and adjusts the learned features into a single channel through convolution, which is then compared with the low angular resolution light field. The conditional graph is obtained by adding the bicubic upsampling results of the angle dimension; Finally, the view-level two-dimensional sine-cosine position encoding is superimposed on the condition map to obtain the angle-aware condition signal c.

7. The method for super-resolution reconstruction of the light field angle under occlusion-sensing condition diffusion according to claim 1, characterized in that, In step S4, the diffusion model includes a forward phase and a reverse phase; The forward phase: random Gaussian noise is gradually added to the low-angle resolution light field to obtain the intermediate state light field, which is called the noisy light field; The reverse stage, also known as the reverse denoising stage, employs a diffusion denoising network for iterative reverse denoising to reconstruct the light field at the target angular resolution. The diffusion denoising network used is DistgU-ViT. The noisy light field and angle sensing condition signal of the current time step are spliced ​​together in the channel dimension and then input into DistgU-ViT. The projected diffused time step is injected into each level of DistgU-ViT, which is the time step embedding. DistgU-ViT outputs noise estimation and determines the noise intensity at the current time step based on a preset diffusion schedule; through iterative denoising, the light field at the target angular resolution is gradually restored. This enables the synthesis of sparse angle views into dense angle views.

8. The method for super-resolution reconstruction of the light field angle under occlusion-sensing condition diffusion according to claim 7, characterized in that, The DistgU-ViT includes an initial convolution, a decoupled residual block, a bottleneck layer, and a lightweight conditional context convolutional network.

9. The method for super-resolution reconstruction of the light field angle under occlusion-sensing conditional diffusion according to claim 1, characterized in that, In step S5, the reverse denoising process is as follows: S41, the noisy light field and angle-sensing condition signal c of the current time step are reconstructed from the sub-aperture image into macro-pixel form, then stitched in the channel dimension, input into DistgU-ViT, and after initial convolution, the channels are adjusted to 64 dimensions to obtain a 64-dimensional signal; S42, the decoupled residual block DisgResBlock performs multi-scale feature extraction on the 64-dimensional signal; the decoupled residual block decouples the coupled optical field information to different domains, including the spatial domain, angular domain, horizontal polar plane domain and vertical polar plane domain. S43, the bottleneck layer uses a Transformer block; the Transformer block includes self-attention, cross-attention, and feedforward networks; wherein, the query of cross-attention comes from the feature token of the current layer; S44, the Conditional Context Convolutional Network downsamples the angle-aware conditional signal c in terms of spatial dimension and transforms it in terms of channel dimension to extract compact semantic conditional tokens; the conditional feature map is compressed by the Conditional Context Convolutional Network to obtain a conditional token sequence; the key and value of the cross attention are derived from the conditional token sequence. S45, based on the noise estimation and diffusion scheduling of DistgU-ViT output, subtract the predicted noise component from the noisy optical field at the current time step to obtain the optical field estimate of the previous time step. S46, repeat S41 to S45, iterating step by step from t=T to... Finally, the target angular resolution light field is obtained. T represents the total training time step. This is the diffusion time step.

Images