Wavelet high frequency hint and residual compensation method for light field spatial super-resolution

Abstract

A wavelet high-frequency cueing and residual compensation method for optical field spatial super-resolution belongs to the field of image spatial super-resolution reconstruction technology. First, geometric fusion features and basic reconstruction results are obtained. Then, a first-order two-dimensional Haar discrete wavelet transform is performed on the geometric fusion features view-by-view to obtain LL, LH, HL, and HH. Finally, the low-frequency structural response of LL is used to guide selective modulation of high-frequency subbands in the LH, HL, and HH directions to form high-frequency cueing features, and controlled residual compensation is performed on the geometric fusion features. This invention reduces noise amplification, false textures, and cross-view edge misalignment; it helps to improve the recovery capability of local textures, text, and fine lines while maintaining the stability of the basic structure.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, computational imaging, and light field image spatial super-resolution reconstruction, specifically to a light field spatial super-resolution method using wavelet high-frequency cueing and residual compensation. Background Technology

[0002] Light field imaging technology can simultaneously record the spatial position and propagation direction information of light rays in a corresponding scene. Compared with traditional two-dimensional imaging, it can provide richer scene structure information for tasks such as digital refocusing, 3D reconstruction, depth estimation, free-viewpoint display, immersive display, and intelligent perception. Typically, light field data can be represented by a sub-aperture image array, where different sub-aperture images correspond to different viewing angles, and each sub-aperture image contains two-dimensional spatial sampling.

[0003] Due to limitations in the total number of pixels in the sensor, the microlens array structure, and the bandwidth of the imaging optical system, there is an inherent contradiction between spatial resolution and angular resolution in the light field acquisition process. As the number of angular samples increases, the number of spatial sampling pixels that can be allocated to each sub-aperture image decreases, resulting in lower spatial resolution of the actually acquired sub-aperture images. This manifests as problems such as blunted edges, blurred text, broken fine lines, unclear texture boundaries, and overly smoothed occluded edges.

[0004] Figure 1 This is a schematic diagram of a light field image space super-resolution reconstruction task. Figure 1 The diagram shows that the number of sub-aperture viewing angles of the input and output light fields remains unchanged, while the spatial size of each sub-aperture image is increased according to a preset magnification. It also shows the structural refinement of the horizontal and vertical epipolar images as the spatial resolution increases. The optical field spatial super-resolution increases the spatial size of each low-resolution sub-aperture image from H×W to sH×sW while keeping the number of U×V sub-aperture viewing angles unchanged. Here, U and V represent the number of viewing angles on the two angular axes, H and W represent the height and width of a single low-resolution sub-aperture image, respectively, and s represents the spatial magnification. Figure 1 The horizontal and vertical epipolar images in the diagram illustrate that this task requires not only recovering the spatial texture and high-frequency structure within a single viewpoint, but also maintaining the geometric correspondence caused by parallax between different viewpoints. If each sub-aperture image is processed independently as a normal two-dimensional image, it may lead to inconsistencies in the positions of edges, textures, or fine lines in adjacent viewpoints, thereby compromising the geometric consistency of the light field image. On the other hand, if there is an over-reliance on cross-viewpoint averaging or global smoothing, it is easy to obtain results with relatively stable geometric relationships but insufficient local high-frequency details.

[0005] Existing light field spatial super-resolution methods typically extract features from sub-aperture images, macro-pixel images, and epipolar images. Convolution-based spatial-angle interaction methods can effectively model local texture and local viewpoint relationships, but are limited by the local receptive field and lack the ability to represent long-range spatial-angle correlations and large parallax geometry. Methods based on self-attention or state-space models enhance long-range dependency modeling capabilities, but usually focus more on overall structure recovery and cross-viewpoint relationship modeling, lacking explicit compensation mechanisms for small-area local high-frequency regions such as text, thin lines, and texture boundaries.

[0006] Furthermore, some methods directly attempt to optimize and enhance the overall high frequency, which can improve local sharpness. However, without reliable cross-viewpoint geometric context constraints, high frequency enhancement may produce inconsistent edge positions and texture responses in different viewpoints, resulting in viewpoint flicker, epipolar breaks, or artifacts. For light field space super-resolution, high frequency restoration is not simply a sharpening problem, but requires controlled local residual compensation on a stable geometric basis.

[0007] From the perspective of light field data organization, sub-aperture image arrays, macropixel images, and epipolar images highlight different physical meanings. Sub-aperture image arrays are convenient for describing spatial texture within a single viewpoint, macropixel images are convenient for aggregating angular information near the same spatial location, and epipolar images can represent cross-viewpoint parallax relationships as a line structure with a certain slope. If light field spatial super-resolution methods rely on only one of these representations, it is usually difficult to simultaneously achieve spatial texture restoration, angular complementarity utilization, and epipolar geometric continuity.

[0008] Traditional single-image super-resolution methods primarily focus on edges, textures, and semantic structures within two-dimensional images, with both input and output typically being a single image. While directly applying such methods to each sub-aperture image can improve local sharpness within a single viewpoint, the independent high-frequency enhancement processes between different viewpoints can easily lead to positional deviations of the same object's edges in adjacent viewpoints, thereby disrupting the inherent cross-viewpoint consistency of the light field data.

[0009] Existing light field super-resolution methods based on space-angle joint learning attempt to introduce angle interaction, macro-pixel rearrangement, or cross-view feature fusion into the network to improve reconstruction quality by utilizing redundant information from multiple views. These methods can improve the inconsistency problem caused by single-view independent super-resolution to some extent, but when there are text, thin lines, fences, occlusion boundaries, or high-contrast textures in the scene, local high-frequency details may still be weakened during multi-view averaging or feature fusion.

[0010] Modeling methods based on epipolar plane images further utilize the line structure within the epipolar plane image (EPI) to represent disparity information, helping to maintain geometric continuity across viewpoints. However, the EPI structure contains both long-distance continuous epipolar lines and local short line segments, abrupt edge changes, and texture breaks. If only long-sequence modeling is used, local details are easily smoothed out during long-range propagation; if only local fragment modeling is used, it is difficult to cover the geometric relationships across the entire viewpoint. Therefore, single-granularity EPI evidence is insufficient to adequately support spatial super-resolution reconstruction of complex regions.

[0011] In recent years, self-attention mechanisms and state-space sequence models have been used in image and light field reconstruction tasks to improve long-range dependency modeling capabilities. Such structures help expand the receptive field and capture global context, but the scanning paths or attention organization methods of general visual backbones do not naturally correspond to the spatial, angular, and epipolar geometric dimensions in the light field. Without topological organization and constraints for the light field, long-range modeling may enhance the overall structural representation, but it still cannot specifically recover high-frequency details in local directions.

[0012] On the other hand, frequency domain loss, wavelet transform, and high-pass filtering are often used to enhance high-frequency information in image super-resolution. However, high-frequency compensation in optical field spatial super-resolution cannot be simply equated with sharpening filtering. For the same physical edge, corresponding geometric positions should be maintained across multiple viewpoints; for occluded and non-Lambertian regions, some high-frequency responses may also exhibit viewpoint dependence. Therefore, high-frequency compensation needs to be based on a reliable geometric context and requires structural guidance mechanisms to control the compensation position, direction, and magnitude.

[0013] In summary, existing technologies still have the following shortcomings: First, although existing spatial, angular, and EPI feature modeling can provide basic geometric representations, there is a lack of clear division of labor between basic structure restoration and local high-frequency detail compensation; second, high-frequency information in the direction of missing text, fine lines, texture boundaries, and occlusion boundaries in the basic reconstruction results lacks explicit candidate representations; third, high-frequency enhancement branches are prone to coupling with the main structure restoration, leading to conflicts between local sharpening and cross-view consistency; and fourth, there is a lack of a compensation mechanism that uses low-frequency structural responses in the feature domain to guide the selection of directional high-frequency subbands and injects them into the basic reconstruction results in the form of controlled residuals.

[0014] Therefore, a spatial super-resolution scheme compatible with existing light field feature modeling backbones is needed. This scheme does not require spatial, angular, or epipolar image modeling itself to be a new invention. Instead, after the light field feature modeling backbone has formed fused features and basic reconstruction results, it further obtains high-frequency directional candidate cues through per-view feature domain wavelet decomposition, selectively modulates high-frequency directional features using low-frequency structural responses, and injects local detail increments into the basic features using a gated small residual method. Simultaneously, it is necessary to construct the basic and final results through shared reconstruction mappings, so that the difference between the two can be explicitly supervised as detail increments generated by the high-frequency compensation branch. Summary of the Invention

[0015] To address the technical problems in existing optical field spatial super-resolution methods, such as insufficient detail recovery in complex high-frequency regions, easy disruption of cross-view consistency by direct high-frequency enhancement, and excessive coupling between basic structure recovery and local detail compensation, this application provides an optical field spatial super-resolution method based on wavelet high-frequency cueing and residual compensation, belonging to the field of optical field image spatial super-resolution reconstruction technology. Geometric fusion features and basic reconstruction results are obtained using an existing or replaceable optical field feature modeling backbone. The geometric fusion features are output by the optical field feature modeling backbone before the high-frequency cueing branch. The basic reconstruction results are obtained by adding the geometric fusion features to a shared reconstruction head and a view-by-view bicubic interpolation baseline. Then, a two-dimensional Haar discrete wavelet transform (DWT) is performed on the geometric fusion features view-by-view to obtain a low-frequency subband LL, a column-direction differential high-frequency subband LH, a row-direction differential high-frequency subband HL, and a row-column joint differential high-frequency subband HH. The low-frequency structural response of LL is used to guide the selective modulation of the high-frequency subbands in the LH, HL, and HH directions, forming high-frequency cueing features, and controlled residual compensation is performed on the geometric fusion features. This invention reduces noise amplification, false textures, and cross-view edge misalignment; it helps to improve the recovery of local textures, text, and fine lines while maintaining the stability of the basic structure.

[0016] A wavelet high-frequency cueing and residual compensation method for optical field spatial super-resolution includes the following steps:

[0017] Step S1, Light field input representation and per-view bicubic interpolation baseline construction:

[0018] Using the low-resolution input light field as the processing object, the light field input representation and view-by-view rearrangement are performed on the sub-aperture image array, and a view-by-view bicubic interpolation baseline is constructed.

[0019] The bicubic interpolation baseline per viewpoint serves as the common pixel domain baseline for subsequent basic reconstruction and final reconstruction;

[0020] Step S2, Light Field Geometric Feature Modeling and Basic Reconstruction:

[0021] Using the light field input representation from step S1 as input, the features output by the light field feature modeling unit are fused to obtain geometric fusion features. These geometric fusion features refer to the light field features formed by fusing view-by-view spatial texture, cross-view correspondence, and epipolar structure information after spatial modeling, angle modeling, and dual-granularity epipolar plane image modeling. The geometric representation indicates the structural correspondence between different viewpoints of the light field, and is not an explicit depth map or 3D geometric map. The geometric fusion features are input to the shared reconstruction head to obtain the basic reconstruction residual. The basic reconstruction residual is added to the view-by-view bicubic interpolation baseline to obtain the basic reconstruction result.

[0022] Step S3, constructing wavelet high-frequency cueing for feature domain per view:

[0023] The geometric fusion features from step S2 are rearranged into per-view SAI features, and a first-level two-dimensional Haar wavelet decomposition is performed on each per-view SAI feature. The low-frequency structural response is used to selectively modulate the high-frequency subbands in different directions to form high-frequency cue features.

[0024] Step S4, gated local residual compensation and final reconstruction:

[0025] The high-frequency cue features from step S3 are used to refine the local residuals of the per-view SAI features corresponding to the geometric fusion features from step S2. The detail compensation amplitude is controlled by gating and residual scaling to obtain the refined features. Then, the shared reconstruction head is added to the per-view bicubic interpolation baseline to obtain the final reconstruction result.

[0026] Step S5, Joint optimization during training phase:

[0027] During the training phase, the basic reconstruction results obtained in step S2, the final reconstruction results obtained in step S4, and the corresponding real high-resolution light field are used to construct a joint loss, and steps S1 to S4 are jointly optimized. During the inference phase, step S5 is not executed, and only steps S1 to S4 are executed sequentially to obtain the final reconstruction results.

[0028] Furthermore, in step S1, the process of representing the light field input and constructing the bicubic interpolation baseline per viewpoint is as follows:

[0029] Step S11, Sub-aperture image array stitching representation and element indexing:

[0030] Let the batch size be B, the number of input luminance channels be 1, the number of viewing angles on the first and second angular axes be U and V respectively, the height and width of each low-resolution sub-aperture image be H and W respectively, and R represent the set of real numbers; the actual input received by the network is a stitched representation of a sub-aperture image (SAI) array. ; The tensor size satisfies: LR indicates low resolution;

[0031] To illustrate the correspondence between the array stitching positions and the physical dimensions of the light field, we introduce the batch index b, the first angular axis index u, the second angular axis index v, the row index y and column index x within the sub-aperture image; and the six-dimensional view tensor of the low-resolution light field. ,but and Space satisfies:

[0032] ;

[0033] Where U, V, H, and W represent the length of the corresponding dimension; u, v, y, and x represent the two angle indices and the row and column indices within the sub-aperture image, respectively, with the indices starting from 0. uH+y determines the vertical position of the y-th row in the array stitching representation within the u-th horizontal view, and vW+x determines the horizontal position of the x-th column in the array stitching representation within the v-th vertical view; this definition avoids mixing dimension length with element index and also clarifies the source of subsequent view-by-view rearrangement;

[0034] Step S12, splitting by viewpoint:

[0035] Will Rearranged into batches consisting of BUV individual sub-aperture images :

[0036] ;

[0037] in, This refers to a general reversible rearrangement operator that splits by viewpoint without changing the element values, only altering the storage layout, rather than being specific to single-channel images. Indicates to Perform the results of the view-by-view split; each batch element corresponds to a specific (u,v) view with a size of 1×H×W; this step ensures that subsequent interpolation and wavelet processing are performed within a single sub-aperture image, avoiding cross-view boundary participation in local operator calculations;

[0038] Step S13, Per-view spatial interpolation and angle arrangement restoration:

[0039] First of all, Each sub-aperture image in the image is subjected to bicubic interpolation using the same spatial magnification s;

[0040] Then, through The inverse rearrangement restores the original U×V angular arrangement, resulting in a bicubic interpolation baseline for each viewpoint. :

[0041] ;

[0042] in, This indicates an operator that performs bicubic interpolation on each sub-aperture image at a magnification of s. This represents the result of magnifying each sub-aperture image in the per-view batch by a factor of s. This represents the inverse rearrangement operator for restoring the U×V viewpoint arrangement. The dimensions are B×1×(UsH)×(VsW); the per-view bicubic interpolation baseline No new viewpoints are generated; only the spatial dimensions of each viewpoint are increased. The basic reconstruction residuals obtained in step S2 and the final reconstruction residuals obtained in step S4 are both consistent with the viewpoint-by-view bicubic interpolation baseline constructed in step S13. Add element by element;

[0043] The per-view bicubic interpolation baseline This serves as the common pixel domain baseline for both the basic reconstruction result of step S2 and the final reconstruction result of step S4.

[0044] Furthermore, in step S2, the process of modeling the optical field geometric features and reconstructing the basic structure is as follows:

[0045] Step S21, Shallow feature mapping:

[0046] First, the low-resolution light field input representation in step S1 is... Expanded along the angular dimension, it becomes a five-dimensional perspective tensor; among which... Using SAI array splicing:

[0047] ;

[0048] Different from The perspective dimension is incorporated into the batch dimension, here. An angular dimension expansion operator; Indicates will In The five-dimensional tensor obtained by arranging the viewpoints into independent viewpoint dimensions;

[0049] Subsequently, a 3D convolution with a kernel size of 1×3×3 was used. Map a single-channel input to aisle, Indicates the number of feature channels;

[0050] Next, through shallow residual mapping Extracting local features :

[0051] ;

[0052] in, express Shallow mapping features after initial 1×3×3 3D convolution; This indicates that the shallow mapping feature has undergone shallow residual mapping. The obtained local residual characteristics; and All have the same dimensions and are added element by element;

[0053] The shallow residual mapping It consists of three consecutive combined layers, each of which includes a 1×3×3 3D convolution and LeakyReLU;

[0054] During training, local features Rearranged by perspective as (BUV) × (HW) × The token sequence is randomly masked by removing 35% of the spatial feature tokens and then the original layout is restored to improve the robustness of the light field feature modeling backbone; random masking is not performed during inference; training enhancement does not change the input and output definitions of subsequent high-frequency cue branches;

[0055] Step S22, Light field feature modeling and backbone feature extraction:

[0056] The backbone is modeled using light field features consistent with the output interface size; in local features Add learnable angle embedding to The rearrangement is (BUV)×H×W× The system is configured to connect 12 light field feature modeling units in series. These 12 light field feature modeling units are divided into 4 groups. Each group contains three units: a SAI modeling unit, a macro-pixel modeling unit, and an EPI modeling unit. The light field feature modeling units are general modeling units in light field image super-resolution and are not limited to any particular modeling method. The light field feature modeling units are configured to use convolution, attention, or state space sequence modeling.

[0057] The recursive process for modeling the light field features is expressed as follows:

[0058] ;

[0059] in, These are the initial characteristics of the skeletal structure; Indicates from B× ×(UV)×H×W to (BUV)×H×W× Layout transformation, For learnable angle embedding, size and same; Output for the nth light field feature modeling unit; This represents the mapping of the nth light field feature modeling unit; It is The result after conversion to a per-view batch layout; the purpose of the light field feature modeling unit is only to generate the geometric fusion features required for subsequent compensation. This does not constitute a necessary structural limitation for the view-by-view wavelet gating and controlled residual compensation in this application;

[0060] Step S23, Multi-level feature aggregation and shallow bypass fusion:

[0061] First, after each light field feature modeling unit is executed, the outputs of the 3rd, 6th, 9th, and 12th units are saved respectively. , , and ;

[0062] Then, output the four groups of data. , , and A multi-level aggregation mapping is constructed by splicing along the channel dimension and performing 3×3 two-dimensional convolution and LeakyReLU. Compress back aisle;

[0063] Subsequently, it was restored to B× The SAI array splicing features of ×(UH)×(VW) and with The restored shallow bypass features are added element by element to obtain several what Fusion features :

[0064] ;

[0065] in, Indicates channel-dimensional splicing. This refers to the process of restoring the feature layout from the perspective of each view to the SAI array stitched feature layout, which is a general term for selecting the corresponding inverse rearrangement according to the input layout. This indicates that shallow features The SAI array splicing features are restored to B×C_f×(U·H)×(V·W);

[0066] In the formula, both addends have been converted to the same B× before addition. The ×(UH)×(VW) layout eliminates the size conflict between the per-view batch and array stitching layouts; the geometric fusion feature It is the SAI array splicing feature output by the light field feature modeling backbone before wavelet high-frequency cueing and controlled residual compensation; the geometric fusion feature only indicates that it has not yet undergone the high-frequency refinement branch, and does not indicate low quality or lack of high frequency;

[0067] Step S24, Reconstructing the Basic Results:

[0068] Shared Reconstruction Head The sequence includes a 1×1 2D convolution, a PixelShuffle with a scaling factor of s, LeakyReLU, and a 3×3 2D convolution; the first convolution will... Channel mapping as The s² channel, the pixel rearrangement upsampling operator (PixelShuffle) rearranges the channel information to the spatial dimension, and finally outputs the single-channel reconstruction residual; the single-channel reconstruction residual is then compared with the per-view bicubic interpolation baseline. The summation yields the basic reconstruction result. :

[0069] ;

[0070] in, Indicates geometric fusion features Input shared reconstruction head The resulting single-channel foundation reconstruction residual has dimensions similar to same;

[0071] The basic reconstruction results It is a geometric fusion feature Without the high-frequency cue feature guidance and controlled residual compensation in steps S3 and S4, and after shared reconstruction head and the view-by-view bicubic interpolation baseline The intermediate reconstruction results obtained by summing are used to restore the main structure, low- and mid-frequency content and basic cross-perspective relationships, but do not mean that they strictly do not contain high frequencies in a frequency sense.

[0072] Furthermore, in step S3, the process of constructing the wavelet high-frequency cue in the per-view feature domain is as follows:

[0073] Geometric fusion features The process involves splitting the data into view-by-view segments and performing a first-level view-by-view two-dimensional Haar-DWT. Three directional gating is generated from the low-frequency subband LL. The modulated column-direction differential high-frequency subband LH, row-direction differential high-frequency subband HL, and row-column joint differential high-frequency subband HH are then transformed using Haar inverse wavelet transform to obtain high-frequency cue features. The low-frequency structural features were reconstructed by LL. For use in step S4;

[0074] Step S31, Geometric fusion feature per-view rearrangement and boundary processing:

[0075] Geometric fusion features Decomposed into BUV per-view SAI features :

[0076] ;

[0077] Each view-by-view SAI feature contains Each channel shares the same set of Haar transform and direction gating parameters; if H or W is odd, then each channel is calculated separately. and , and These represent the fill amounts in the height and width directions, respectively, and are only added on the right side. Column, supplement below Okay, use reflective fill. For reflection filling operators, This indicates the view-by-view features after reflection filling:

[0078] ;

[0079] When both H and W are even numbers = =0, no padding is performed; after performing the inverse Haar transformation in step S34, the result is cropped to the original H×W, and the padded size should not be mistaken for the module output size;

[0080] Step S32, First-order per-view 2D Haar-DWT:

[0081] Step S321, odd / even position sub-sampling:

[0082] After filling Sampling is performed in both row and column directions according to odd and even positions, resulting in four sets of tensors:

[0083] ;

[0084] ;

[0085] in, Corresponding to even rows and even columns, Corresponding to even rows and odd columns, Corresponding to odd-numbered rows and even-numbered columns, The four tensors have the same size; i and j represent the row and column indices in the Haar subband, respectively, and are both integers starting from 0.

[0086] Step S322: Calculate the four Haar subbands using an orthogonal two-dimensional Haar transform coefficient of 0.5.

[0087] ; ;

[0088] ; ;

[0089] Wherein, LL is the low-frequency subband; LH is the column-direction differential high-frequency subband; HL is the row-direction differential high-frequency subband; HH is the row-column joint differential high-frequency subband; LL is the low-frequency structural response in the local average sense; LH is sensitive to the differential of adjacent columns in the same row, HL is sensitive to the differential of adjacent rows in the same column, and HH is sensitive to the joint changes of rows and columns.

[0090] Step S323, Dimension Recording:

[0091] The spatial dimensions of the four Haar subbands are all Simultaneously, the original dimensions (H, W) before wavelet transform are recorded for subsequent inverse transform pruning; the two-dimensional Haar wavelet decomposition is applied to the view-by-view feature map that already contains the basic optical field context. It acts on the input pixel image, rather than directly on the entire light field sub-aperture image array mosaic;

[0092] Step S33, three-way high-frequency gating guided by low-frequency structure:

[0093] Step S331, Gated Tensor Generation:

[0094] Input LL into a lightweight gated network consisting of 1×1 convolutions, GELU, 1×1 convolutions, and a Sigmoid function; the first convolution holds... Channel, second convolution output aisle, Represents the Sigmoid function, gated tensor It is expressed as follows:

[0095] ;

[0096] in, This indicates that both the number of input and output channels are 1. The first 1×1 two-dimensional convolution; Indicates will Channel mapping is 3 The second 1×1 two-dimensional convolution of the channel; This represents the result after LL is processed by two convolutions and an intermediate GELU;

[0097] Step S332, Directional Splitting:

[0098] Along the channel Divide into three groups on average: ,and The three sets of gating tensors correspond one-to-one with LH, HL, and HH, respectively, and all have the shape of LH, HL, and HH. ;

[0099] Step S333, directional modulation centered on the identity mapping:

[0100] set up This represents the original high-frequency subband in direction d. For directional gating, a learnable scalar is used. The following operations are performed to obtain the modulated subband. :

[0101] ;

[0102] in, For the original high-frequency subband in direction d, For corresponding gating, For the modulated subband, ⊙ represents element-wise multiplication, and the parameter... Initialized to 0, therefore, at the beginning of training, the modulation branch does not change the original high-frequency subband; as training progresses, Together with three directional gating, it learns to slightly enhance or suppress the response in the corresponding direction under a given LL structure response; the above modulation method is different from directly multiplying the high-frequency subband with the 0 to 1 Sigmoid result, which can avoid compressing the high frequency uniformly to about half in the initial state;

[0103] Step S34, reconstruction of high-frequency cue features and low-frequency structural features:

[0104] Construct a zero tensor with the same shape as LL: Set the low-frequency input to zero, perform an inverse Haar transform on the three modulated high-frequency subbands, and clip them back to H×W to obtain the high-frequency cue features. :

[0105] ;

[0106] in, The high-frequency subbands of the three modulated directions are reconstructed only as high-frequency cue features, rather than directly as pixel-domain super-resolution results; This indicates that the right and bottom fill generated in step S31 will be trimmed. This represents the two-dimensional Haar inverse wavelet transform; simultaneously, the LL subband is used as the only non-zero subband to perform the inverse transform and is then pruned to obtain the low-frequency structural features. :

[0107] ;

[0108] in, The low-frequency structural features are obtained by reconstructing the LL low-frequency subband. Used for subsequent low-frequency structure gating; and The dimensions are all with same, , and All three participate in residual compensation without changing the number of viewpoints or spatial layout;

[0109] Therefore, the output of step S3 is... High-frequency cue features of the same size and low-frequency structural features This is for use in step S4.

[0110] Furthermore, in step S4, the gated local residual compensation and final reconstruction process is as follows:

[0111] Step S41, High-frequency cue feature-driven local residual prediction:

[0112] First, the view-by-view SAI features obtained in step S31 are... High-frequency cue features obtained in step S34 By splicing along the channel dimension, the fused feature Q is obtained:

[0113] ;

[0114] Step S411, Channel Compression:

[0115] Use 1×1 convolution Put Q's Channel compression to Channels for fusing per-view SAI features and high-frequency prompt features ;

[0116] Step S412, candidate residual output:

[0117] The features compressed in step S411 are sequentially input into four cascaded lightweight local convolution residual blocks for local feature transformation, and then subjected to a 1×1 two-dimensional convolution at the end. Obtaining and per-view SAI features Candidate residuals of the same size :

[0118] ;

[0119] in, This represents four lightweight locally convolutional residual blocks. The first sub-block of each block sequentially performs Layer Normalization (LayerNorm), 1×1 channel expansion, 3×3 depthwise convolution, channel binary multiplication gate (SimpleGate), and 1×1 channel restoration, followed by residual concatenation with zero initialization. The second sub-block sequentially performs LayerNorm, 1×1 expansion, SimpleGate, and 1×1 restoration, followed by residual concatenation with zero initialization. This implementation does not use global pooling channel attention, aiming to preserve the local patching property and make the initialization of new branches close to identity.

[0120] Step S42, generation of detail gating and low-frequency structure gating:

[0121] Two sets of independent 1×1 convolutions are used to generate detail gating. and low-frequency structure gating :

[0122] ;

[0123] in, and These represent two independent 1×1 two-dimensional convolutions. Represents the Sigmoid function; detail gating SAI features from each perspective and high-frequency prompt features The concatenated Q is used as input to determine the local injection location of the candidate residuals; low-frequency structure gating. The same-sized structural features reconstructed by LL are used to further adjust the residual injection amplitude; detail gating. and low-frequency structure gating Because of their different functions, they cannot be simply replaced by a single undefined gating mechanism; detailed gating... and low-frequency structure gating In step S43, the control is further combined to form a joint gating system;

[0124] Step S43, combined gating and small-scale residual injection:

[0125] First, use learnable scalars Low-frequency structure gating Convert to a 1-centered adjustment factor and multiply it element-wise with the detail gating:

[0126] ;

[0127] in, Indicates joint gating, For low-frequency structure-gated scaling scalars, initialized to 0, therefore at the start of training... equal Low-frequency gating remains neutral;

[0128] Then, learnable small-scale residual proportions were used. Controlling the overall magnitude of candidate residuals:

[0129] ;

[0130] in, The scaling factor is the overall residual; therefore, the local residual refinement branch in step S4 only applies the view-by-view SAI features during the early stages of training. Make minor corrections, and then learn appropriate gating and residual amplitude based on the monitoring signal to reduce the risk of high-frequency enhancement damaging the existing foundation structure;

[0131] Step S44, refine feature recovery and shared reconstruction:

[0132] Will Restore the SAI array stitching features according to the original U×V viewpoint order to obtain the array stitching features. :

[0133] ;

[0134] in, The dimensions are B× ×(UH)×(VW);

[0135] Will Input to the same shared reconstruction head described in step S24 and the same as obtained in step S13 Add them together to get the final reconstruction result. :

[0136] ;

[0137] in, To ultimately reconstruct the residuals;

[0138] Geometric fusion features With refined features All input parameters are shared by the same reconstruction head and respectively with the same viewpoint bicubic interpolation baseline The results are added together; therefore, the difference between the two reconstruction results mainly comes from the characteristic differences before and after controlled residual compensation.

[0139] Step S45, Construction of pixel domain output increment in training state:

[0140] Output during training and And construct pixel-domain detail increment :

[0141] ;

[0142] Where stopgrad represents an operator that keeps the value unchanged but blocks gradient backpropagation; The controlled refinement branch is on the same viewpoint bicubic interpolation baseline. Reconstruction head with shared parameters The pixel-domain detail increment caused by the reconstruction head; since the reconstruction head includes convolution, pixelShuffle and non-linear activation, the increment is not mathematically guaranteed to be a pure high frequency; during the training phase, the high-pass loss in step S54 guides it to focus mainly on local details that have not yet been recovered from the basic results;

[0143] Step S46, inference state output:

[0144] High-resolution labels are not required during inference, and the underlying reconstruction results are not used. and pixel domain detail increment Used for loss calculation; during the inference phase, steps S1 to S4 are executed sequentially and the final reconstruction result is output. If the system requires diagnostics or visualization, then retain [this option]. It serves as an auxiliary output, but does not affect the final reconstruction process.

[0145] Furthermore, in step S5, the joint optimization process during the training phase is as follows:

[0146] Step S51, pixel reconstruction loss:

[0147] Let the stitching of a real high-resolution light field sub-aperture image array be represented as: Size and and The same applies, where HR represents high resolution and SR represents super-resolution. Represents a true high-resolution light field. This indicates the final super-resolution reconstruction result, which is the final reconstruction result. and The mean absolute errors are as follows:

[0148] ;

[0149] in, For the necessary main reconstruction supervision; To assist in supervision; mean indicates averaging over all specified elements of the tensor;

[0150] Step S52, frequency radius-weighted view-by-view Fourier loss:

[0151] First and The images are rearranged into (BUV)×1×(sH)×(sW) per-view batches, and then orthogonally normalized two-dimensional real fast Fourier transform rfft2 is used to obtain the final reconstructed semispectral result. Semi-spectrum of the actual results Construct weights based on the normalized frequency radius:

[0152] ;

[0153] in, Represents the normalized frequency radius. Indicates the frequency radius weight. These represent the vertical and horizontal frequency coordinates, respectively. `fftfreq` is used to generate the complete frequency axis coordinates. This represents the maximum frequency radius of the half-spectrum grid. Indicates frequency emphasis index, Generated by fftfreq, rfftfreq is used to generate the real Fourier half-spectrum frequency axis coordinates. Generated by rfftfreq; weights are then divided by their own mean to make the loss scale more stable for different image patch sizes; weighted absolute errors are calculated for the real and imaginary parts of the complex spectrum respectively:

[0154] ;

[0155] in, and These represent the real and imaginary parts of a complex number, respectively. FFT stands for Fast Fourier Transform. The loss reduces the relative weight of low-frequency components and supplements the constraints of pixel L1 on mid-to-high-frequency content.

[0156] Step S53, two-level per-view Haar wavelet loss for edge sensing:

[0157] First, use horizontal and vertical Sobel kernels. , Edge responses are generated on real high-resolution per-view images and normalized within each sample:

[0158] ;

[0159] in, Represents the edge response of a real image; Indicates the normalized edge weights; and These are horizontal and vertical Sobel kernels, respectively. This represents the edge emphasis coefficient, which enables... Stop the gradient and scale to the appropriate subband size using bilinear interpolation; and Perform the first level of Haar-DWT, comparing LH, HL, and HH; then perform separate operations on... and The first-stage LL continues with the second-stage Haar-DWT, comparing the three high-frequency subbands of the second stage; let the set of directional high-frequency subbands D = {LH, HL, HH}, and the loss... for:

[0160] ;

[0161] in, Index for the direction subband; express In the direction of level 1 High-frequency subband; express In the direction of level 2 High-frequency subband; express The corresponding subband; express The corresponding subband; and These are the edge weights scaled to the two-level sub-band size; β is the second-level relative weight; the first-level DWT in the main branch of step S3 acts on the geometric fusion feature. Used to generate high-frequency cue features The two-stage DWT in this step operates on the pixel domain. and , used for loss calculation;

[0162] Step S54, pixel domain detail increment Qualcomm loss:

[0163] Make the target residual Define a high-pass operator that uses k×k average pooling after reflection filling, and compare the high-pass components of the predicted increment and the target residual:

[0164] ;

[0165] in, Represents any input tensor of a high-pass operator. Indicates reflection filling. This represents k×k average pooling. Indicates the corresponding high-pass operator; Indicates pixel-level detail increment;

[0166] Target residual The increment represents the unrecovered pixel difference between the true image and the base result. L1 is calculated after passing through the same high-pass operator on both sides, which can reduce the interference of low-frequency brightness differences on detail supervision.

[0167] Step S55, epipolar plane gradient consistency and angle difference consistency:

[0168] The high-resolution light field sub-aperture image array is stitched together and represented as B×1×U×V× × ,in =sH、 =sW; Fix the horizontal EPI to (v,y) and observe the (u,x) plane, fix the vertical EPI to (u,x) and observe the (v,y) plane, batch layout. for:

[0169] ;

[0170] in, Represents high-resolution light field data; For horizontal EPI rearrangement operators; For the vertical EPI rearrangement operator, calculate the first-order difference for each EPI tensor along the last spatial axis and the penultimate angular axis, and average the L1 gradient errors of the two axes, denoted as . The horizontal and vertical EPI losses are then averaged to obtain the epipolar plane gradient consistency loss L. epi :

[0171] ;

[0172] represent the horizontal EPI tensors obtained from the final reconstruction result and the true high-resolution light field through horizontal EPI rearrangement, respectively; These represent the vertical EPI tensors obtained from the final reconstruction result and the true high-resolution light field through vertical EPI rearrangement, respectively.

[0173] Step S56, Preheating and Joint Weighting:

[0174] Let the current training round be e, and the auxiliary loss warm-up rounds be e. ,definition:

[0175] ;

[0176] in, Let e ​​represent the auxiliary loss weight in round e. When e=1, the warm-up weight is 0, then it increases linearly and is truncated to 1; let the auxiliary loss index set be... Each element corresponds to the frequency domain, wavelet, increment, EPI, and optional angle difference loss described in steps S52 to S55, respectively. The total loss is then calculated as follows: Represented as:

[0177] ;

[0178] in, Indicates the auxiliary loss category, This indicates the weight corresponding to the auxiliary loss category;

[0179] Step S57, parameter update:

[0180] Inputting low-resolution light field representation Enter steps S1 to S4 sequentially to obtain... , and The enabled loss term is calculated and summed according to steps S51 to S56. The parameter gradient is obtained through backpropagation. Then, the configured gradient optimizer is used to update the light field feature modeling backbone, orientation gating, local residual refinement branch and shared reconstruction head.

[0181] Step S58, the relationship between the training and reasoning stages:

[0182] Training phase and As input, steps S1 to S5 are executed to determine each learnable parameter; the parameters are fixed during the inference phase, using only... Execute steps S1 to S4 as input and output. .

[0183] An electronic device, characterized in that it comprises: one or more processors; a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the above-described optical field spatial super-resolution method.

[0184] A computer-readable storage medium, characterized in that the computer-readable storage medium stores program code, which can be invoked by a processor to execute the above-described optical field spatial super-resolution method.

[0185] Compared with the prior art, the technical effects of the present invention are as follows:

[0186] (1) This invention explicitly decouples the basic structure restoration from the local high-frequency compensation, so that the basic reconstruction result mainly undertakes the restoration of the main structure, low- and mid-frequency content and basic cross-view consistency, while the high-frequency branch focuses on compensating for local details such as text, fine lines, texture boundaries and occlusion boundaries that are still missing in the basic result;

[0187] (2) The present invention utilizes the low-frequency structural response of LL to perform gated modulation on the high-frequency subbands in the LH, HL and HH directions, so that the high-frequency compensation has the ability to select the direction and guide the structure, avoiding the indiscriminate injection of all high-frequency responses, thereby reducing noise amplification, false texture and cross-view edge misalignment;

[0188] (3) The present invention performs two-dimensional wavelet decomposition in the feature domain rather than the image domain, and uses high-frequency information as cue features to participate in residual refinement, instead of directly sharpening the output image, which helps to improve the recovery ability of local textures, text and fine lines while maintaining the stability of the basic structure.

[0189] (4) This invention decouples the basic reconstruction result from the final reconstruction result by residual decoupling constraint, so that the high-frequency branch mainly learns the local detail increments missing in the basic result, restricts its unconstrained modification of the whole light field image, and improves the training stability and result interpretability.

[0190] (5) The modular organization of the present invention has good scalability. The number of viewpoints, spatial magnification, state space modeling unit, self-attention unit, convolution thinning unit and upsampling unit can all be replaced or adjusted according to actual needs, but the core process of feature domain wavelet high-frequency prompts, low-frequency structure guidance direction high-frequency selection, shared reconstruction mapping of basic reconstruction results and final reconstruction results and pixel domain detail incremental supervision is still maintained. Attached Figure Description

[0191] Figure 1 This is a schematic diagram of a light field image spatial super-resolution reconstruction task.

[0192] Figure 2 This is a flowchart illustrating the overall process of light field image spatial super-resolution reconstruction according to the present invention.

[0193] Figure 3 This is a schematic diagram of the feature domain wavelet high-frequency prompting and controlled residual compensation module. Detailed Implementation

[0194] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0195] The embodiments described herein are intended to illustrate possible implementations of the invention, and not to limit the invention to a specific light field backbone, a specific training dataset, or a specific combination of parameters. Unless otherwise stated, this embodiment uses a 5×5 input viewpoint, 4x spatial super-resolution, and luminance Y-channel reconstruction as an example; for a color light field, this method can be applied to the Y-channel, and the Cb and Cr chromaticity component channels are interpolated to the target spatial size viewpoint by viewpoint before being combined with the reconstructed Y-channel to synthesize the color result. In this embodiment, both the network input and output are single-channel luminance components; the RGB three channels are not directly input into the network.

[0196] like Figure 2 As shown, the low-resolution light field is subjected to shallow feature mapping, basic light field backbone modeling and multi-level feature fusion to obtain geometric fusion features; the geometric fusion features are used to generate basic reconstruction results through a shared reconstruction head, and after feature domain wavelet high-frequency cueing and controlled residual compensation, the final reconstruction result is generated. The basic reconstruction result and the final reconstruction result are both added to the same viewpoint bicubic interpolation baseline.

[0197] A wavelet high-frequency cueing and residual compensation method for optical field spatial super-resolution firstly normalizes the input low-resolution optical field and constructs a view-by-view upsampling baseline; then, it extracts geometric fusion features to obtain the basic reconstruction result; performs two-dimensional wavelet decomposition on the geometric fusion features view-by-view, and uses low-frequency structural response to guide high-frequency subband modulation to form local high-frequency cueing; based on the local high-frequency cueing, it performs gated local residual compensation on the geometric fusion features to obtain the final reconstruction result; and uses the basic reconstruction result, the final reconstruction result, and the corresponding real high-resolution optical field to construct a joint optimization objective.

[0198] Step S1, Light field input representation and per-view upsampling baseline construction: Taking the input low-resolution light field as the processing object, the sub-aperture image array is stitched and rearranged per-view, and a per-view upsampling baseline is constructed. The upsampling baseline serves as the common pixel domain baseline for subsequent basic reconstruction and final reconstruction.

[0199] Step S2, geometric fusion feature extraction and basic reconstruction: Using the input light field representation in step S1 as input, geometric fusion features are obtained through basic feature modeling and multi-level feature fusion, and the basic reconstruction result is obtained by using the shared reconstruction head in combination with the per-view upsampling baseline;

[0200] Step S3, constructing wavelet high-frequency cues in the feature domain per-view: using the geometric fusion features obtained in step S2 as input, rearrange them into per-view feature maps, and perform two-dimensional Haar wavelet decomposition on each per-view feature map. Use the low-frequency structural response to selectively modulate the high-frequency subbands in different directions to form local high-frequency cues.

[0201] Step S4, gated local residual compensation and final reconstruction: The local high-frequency cue obtained in step S3 is used to refine the local residual of the geometric fusion feature in step S2. The detail compensation amplitude is controlled by gating and residual scaling to obtain the refined feature. The final reconstruction result is obtained by using the same shared reconstruction head as in step S2 combined with the per-view upsampling baseline constructed in step S1.

[0202] Step S5, Joint optimization during training: During the training phase, the basic reconstruction results obtained in step S2, the final reconstruction results obtained in step S4, and the corresponding real high-resolution light field are used to construct a joint loss and jointly optimize the aforementioned modules; during the inference phase, step S5 is not executed, and only steps S1 to S4 are executed sequentially to obtain the final reconstruction results.

[0203] Furthermore, in step S1, the process of representing the light field input and constructing the per-view upsampling baseline is as follows:

[0204] Step S11, Sub-aperture image array stitching representation and element indexing:

[0205] Let the batch size be B, the number of input luminance channels be 1, the number of viewing angles on the first and second angular axes be U and V respectively, the height and width of each low-resolution sub-aperture image be H and W respectively, and R represent the set of real numbers. The actual input received by the network is a stitched representation of a sub-aperture image (SAI) array. Its tensor size is:

[0206] ;

[0207] In this embodiment, U=V=5. To illustrate the correspondence between the array stitching position and the physical dimensions of the light field, the batch index b, the first angular axis index u, the second angular axis index v, the row index y and column index x within the sub-aperture image are introduced. Let... The six-dimensional view tensor representing the low-resolution light field is then... splicing elements with the actual input array The following conditions must be met:

[0208] ;

[0209] The lowercase letters above only indicate the index, while uppercase U, V, H, and W indicate the corresponding dimension length; the index uses a 0-based starting convention.

[0210] ;

[0211] Therefore, uH+y determines the vertical position of the y-th row within the u-th horizontal viewpoint in the array stitching representation, and vW+x determines the horizontal position of the x-th column within the v-th vertical viewpoint in the array stitching representation. This definition avoids confusing dimension length with element index and clarifies the source of subsequent viewpoint-by-view rearrangement.

[0212] Step S12, splitting by viewpoint:

[0213] Will The batch is rearranged into BUV images of individual sub-apertures:

[0214] ;

[0215] in, This represents a general, reversible rearrangement operator that splits elements by viewpoint without changing their numerical values, only altering the storage layout, rather than being specific to single-channel images. Each batch of elements corresponds to a defined (u,v) viewpoint with dimensions of 1×H×W. This step ensures that subsequent interpolation and wavelet processing are performed within a single sub-aperture image, avoiding cross-viewpoint boundaries from participating in local operator calculations.

[0216] Step S13, Per-view spatial interpolation and angle arrangement restoration:

[0217] right Each sub-aperture image in the image undergoes bicubic interpolation using the same spatial magnification s, in this embodiment s=4; subsequently, through The inverse rearrangement restores the original U×V angular arrangement, yielding a high-resolution interpolation baseline:

[0218] ;

[0219] in, This indicates an operator that performs bicubic interpolation on each sub-aperture image at a magnification of s. This represents the inverse rearrangement operator for restoring the U×V viewpoint arrangement. The dimensions are B×1×(UsH)×(VsW). This baseline does not generate new viewpoints, but only increases the spatial dimensions of each viewpoint. Both the basic reconstruction branch and the final reconstruction branch of the model are added to the same baseline, so the comparison between the two results is not affected by different interpolation methods.

[0220] Furthermore, in step S2, the geometric fusion feature extraction and basic reconstruction process is as follows:

[0221] Step S21, Shallow feature mapping:

[0222] The input sub-aperture image array is stitched together and expanded into a five-dimensional view tensor along the angular dimension:

[0223] ;

[0224] Different from The perspective dimension is incorporated into the batch dimension, here. The U×V sub-aperture views in the array stitching representation are arranged into independent view dimensions, while keeping the batch dimension B unchanged, thus obtaining a five-dimensional view tensor with a view depth of UV. Subsequently, a three-dimensional convolution with a kernel size of 1×3×3 is used. Map a single-channel input to Channel; This embodiment =64. Then, a shallow residual mapping consisting of three 1×3×3 3D convolutions and LeakyReLU is applied. Extracting local features yields:

[0225] ;

[0226] During training, you can Rearranged by perspective as (BUV) × (HW) × The token sequence is randomly masked by removing 35% of the spatial feature tokens before restoring the original layout to improve the robustness of the light field feature modeling backbone; this random masking is not performed during inference. This training enhancement does not change the input and output definitions of subsequent high-frequency cue branches.

[0227] Step S22, Basic light field backbone feature extraction:

[0228] This invention allows for the modeling of the backbone using light field features with consistent output interface dimensions. One embodiment is... Add learnable angle embedding to The rearrangement is (BUV)×H×W× It is implemented by connecting 12 light field feature modeling layers in series. These 12 layers are divided into 4 groups, each containing three light field feature modeling units; in the implementation, each group sequentially employs a spatial modeling module, a joint light field modeling module, and a dual-granularity EPI modeling module. Its recursive process can be expressed as:

[0229] ;

[0230] in, Indicates from B× ×(UV)×H×W to (BUV)×H×W× Layout transformation, This represents the nth light field feature modeling layer. The aforementioned light field feature modeling layer can be modeled using convolution, attention, or state space sequence. Its purpose is only to generate the geometric fusion features required for subsequent compensation and does not constitute a necessary structural constraint for the view-by-view wavelet gating and controlled residual compensation of this application.

[0231] Step S23, Multi-level feature aggregation and shallow bypass fusion:

[0232] After completing the basic modeling of each group of three layers, save the files respectively. , , and The four sets of hierarchical features are concatenated along the channel dimension and then subjected to a multi-level aggregation mapping consisting of 3×3 two-dimensional convolution and LeakyReLU. Compress back Channel; subsequently restored to B× The SAI array splicing features of ×(UH)×(VW) and with The restored shallow bypass features are summed element by element:

[0233] ;

[0234] In the formula, Indicates channel-dimensional splicing. This represents the recovery from a per-view feature layout to a SAI array stitched feature layout. In the formula, both addends have been converted to the same B× value before addition. The ×(UH)×(VW) layout avoids the size conflict caused by directly adding up the per-view batch and array stitching layouts. This is called geometric fusion feature, which is the SAI array splicing feature output by the basic optical field backbone before wavelet high-frequency cueing and controlled residual compensation; "basic" only means that it has not yet undergone the high-frequency refinement branch, and does not mean low quality or lack of high frequency.

[0235] Step S24, Reconstructing the Basic Results:

[0236] Shared Reconstruction Head The sequence includes a 1×1 2D convolution, a PixelShuffle with a scaling factor of s, LeakyReLU, and a 3×3 2D convolution. The first convolution will... Channel mapping as The s² channel, using the PixelShuffle upsampling operator, rearranges the channel information to the spatial dimension, finally outputting a single-channel reconstruction residual. This residual is then compared with the per-view bicubic interpolation baseline. The summation yields the basic reconstruction result:

[0237] ;

[0238] The basic reconstruction results It is a geometric fusion feature. Without the high-frequency prompts and controlled residual compensation in steps S3 and S4, and after shared reconstruction head and chase The intermediate reconstruction result obtained by addition is used to undertake the restoration of the main structure, low- and mid-frequency content and basic cross-perspective relationships, but it does not mean that it strictly does not contain high frequency in a frequency sense.

[0239] The process includes view-by-view splitting of geometric fusion features, view-by-view two-dimensional Haar wavelet decomposition, low-frequency subband-guided high-frequency subband modulation, inverse wavelet transform high-frequency cue reconstruction, gated local residual compensation, and view arrangement restoration, such as... Figure 3As shown; Step S3 decomposes the geometric fusion features from one viewpoint and performs two-dimensional Haar-DWT, generates three-directional gating by LL, modulates LH, HL, and HH, and obtains high-frequency cue features through inverse wavelet transform. The low-frequency structural features were reconstructed by LL. For use in step S4;

[0240] Step S31, Geometric fusion feature per-view rearrangement and boundary processing:

[0241] Will Decomposed into BUV per-view feature maps:

[0242] ;

[0243] Each view feature contains Each channel shares the same wavelet and local refinement parameters. If H or W is odd, then each channel is calculated separately. and Only on the right side Column, supplement below Okay, use reflective fill. Fill operator for reflection:

[0244] ;

[0245] When both H and W are even numbers = =0, no padding is performed. This embodiment typically uses 32×32 low-resolution image patches, so padding is usually unnecessary; this processing is still retained to support general sizes. After the inverse transformation, the image must be cropped back to its original H×W size; the padded size should not be mistaken for the module's output size.

[0246] Step S32, perspective-by-view two-dimensional Haar wavelet decomposition:

[0247] Step S321, odd / even position sub-sampling:

[0248] After filling Sampling is performed in both row and column directions according to odd and even positions, resulting in four sets of tensors:

[0249] ;

[0250] ;

[0251] in, Corresponding to even rows and even columns, Corresponding to even rows and odd columns, Corresponding to odd-numbered rows and even-numbered columns, Corresponding to odd-numbered rows and columns; all four have the same size.

[0252] Step S322, calculate the four Haar subbands:

[0253] Calculated using orthogonal two-dimensional Haar transform coefficients of 0.5:

[0254] ; ;

[0255] ; ;

[0256] LL represents the low-frequency structural response in a locally average sense; LH is more sensitive to the difference between adjacent columns within the same row, HL is more sensitive to the difference between adjacent rows within the same column, and HH is more sensitive to the combined row and column changes. To avoid the conflicting naming conventions of "horizontal high frequency" and "vertical edge" in different literature, this embodiment uses "column direction difference response," "row direction difference response," and "combined row and column difference response" to describe the three high-frequency subbands.

[0257] Step S323, Dimension Recording:

[0258] The spatial dimensions of all four sub-bands are Simultaneously, the original dimensions (H, W) before wavelet transform are recorded for subsequent inverse transform pruning. The DWT operates on view-by-view features that already contain the underlying optical field context. It acts on the input pixel image, rather than directly on the entire light field sub-aperture image array mosaic.

[0259] Step S33, three-way high-frequency gating guided by low-frequency structure:

[0260] Step S331, Gated Tensor Generation:

[0261] Input LL into a lightweight gated network consisting of 1×1 convolutions, GELU, 1×1 convolutions, and a Sigmoid function. The first convolution maintains... Channel, second convolution output aisle, The Sigmoid function can be represented by the following formula:

[0262] ;

[0263] Step S332, Directional Splitting:

[0264] Along the channel Divide into three groups on average: ;

[0265] The three sets of gates correspond one-to-one with LH, HL, and HH, and all have the following shapes. .

[0266] Step S333, directional modulation centered on the identity mapping:

[0267] make Indicates the original directional sub-band. For directional gating, a learnable scalar is used. The following operations are performed to obtain the modulated subband. :

[0268] ;

[0269] parameter Initialized to 0, all three multiplication factors are 1 at the start of training, and the modulation branch does not change the original high-frequency subband; as training progresses... Together with three directional gating mechanisms, the system learns to slightly enhance or suppress the response in the corresponding direction given the LL structure response. This design differs from directly multiplying the high-frequency subband by the 0-1 Sigmoid result, thus avoiding the initial state uniformly compressing the high frequencies to about half.

[0270] Step S34, High-frequency cue and low-frequency structural feature reconstruction:

[0271] Construct a zero tensor with the same shape as LL. Set the low-frequency input to zero, perform an inverse Haar transform on the three modulated high-frequency subbands, and clip them back to H×W to obtain:

[0272] ;

[0273] Should Reconstruction is performed using only three high-frequency sub-bands in three directions as candidate high-frequency cues, rather than directly as pixel-domain super-resolution results. This indicates that the right / bottom padding generated in step S31 has been removed. Simultaneously, LL is treated as the only non-zero subband, and an inverse transformation is performed followed by clipping, resulting in:

[0274] ;

[0275] Used for subsequent low-frequency structure gating. and The dimensions are all with Similarly, all three can participate in residual compensation without changing the number of viewpoints or spatial layout.

[0276] Furthermore, in step S4, the gated local residual compensation and final reconstruction process is as follows:

[0277] Step S41, High-frequency cue-driven local residual prediction:

[0278] First, the original view-by-view basic features are... High-frequency prompt features In channel splicing:

[0279] ;

[0280] Step S411, Channel Compression:

[0281] Use 1×1 convolution Put Q's Channel compression to Channels are used to integrate original features and wavelet high-frequency cues.

[0282] Step S412, Lightweight local feature transformation:

[0283] This embodiment uses four lightweight local convolutional residual blocks sequentially, denoted as... The first sub-block of each block sequentially performs Layer Normalization (LayerNorm), 1×1 channel expansion, 3×3 depthwise convolution, channel binary multiplication gating (SimpleGate), and 1×1 channel restoration, followed by residual connections with zero initialization. The second sub-block sequentially performs LayerNorm, 1×1 expansion, SimpleGate, and 1×1 restoration, followed by residual connections with zero initialization. This implementation does not use global pooling channel attention, aiming to preserve local patching properties and make the initialization of new branches close to identity.

[0284] Step S413, candidate residual output:

[0285] 1×1 convolution at the end Get and Candidate residuals of the same size:

[0286] ;

[0287] Step S42, generation of detail gating and low-frequency structure gating:

[0288] Two sets of independent 1×1 convolutions are used to generate detail gating and low-frequency structure gating, respectively, as follows:

[0289] ;

[0290] Detail gating observes both geometric fusion features and high-frequency cue features to determine the local injection location of candidate residuals; low-frequency structure gating observes structural features of the same size reconstructed by LL to further adjust the injection amplitude. Because these two functions differ, they cannot be replaced by a single undefined gating mechanism. Here, Q is the feature obtained in step S41 by stitching together the per-view basic features and high-frequency cue features along the channel dimension. The low-frequency structural features obtained from the LL low-frequency subband reconstruction in step S34; and These represent two independent 1×1 two-dimensional convolutions. Represents the Sigmoid function. Details gating. Local injection for adjusting candidate residuals, low-frequency structure gating The two are used to further adjust the injection amplitude, and are further combined in step S43 to form a joint gating.

[0291] Step S43, combined gating and small-scale residual injection:

[0292] First, use learnable scalars The low-frequency structure gate is converted into a regulation factor centered at 1, and then multiplied element-wise with the detail gate:

[0293] ;

[0294] As a low-frequency structure-gated scaling scalar, it is initialized to 0, so G equals 0 at the start of training. Low-frequency gating is kept neutral. Then, a learnable small-scale residual ratio is used. Controlling the overall magnitude of candidate residuals:

[0295] ;

[0296] In this embodiment The initial value is 0.01. Therefore, the new branch only makes minor modifications to the basic features in the early stages of training, and then learns appropriate gating and residual amplitude based on the supervision signal, reducing the risk of high-frequency enhancement damaging the existing basic structure.

[0297] Step S44, refine feature recovery and shared reconstruction:

[0298] Will Restore the SAI array stitching features according to the original U×V viewpoint order:

[0299] ;

[0300] The size of this feature is B× ×(UH)×(VW). Input it into the same shared reconstruction head described in step S24. and the same as obtained in step S13 Add:

[0301] ;

[0302] Geometric fusion features With refined features All input parameters are shared by the same reconstruction head and respectively with the same viewpoint bicubic interpolation baseline Therefore, the difference between the two reconstruction results mainly comes from the characteristic differences before and after controlled residual compensation.

[0303] Step S45, Construction of pixel domain output increment in training state:

[0304] Output during training , And construct:

[0305] ;

[0306] `stopgrad` indicates stopping at this difference objective. The gradient on one side. This is the pixel-domain output increment caused by the controlled thinning branch under the same baseline and the same reconstruction head. Since the reconstruction head includes convolution, pixelShuffle, and non-linear activation, this increment is not mathematically guaranteed to be purely high-frequency; during the training phase, it is guided by the high-pass loss in step S54 to focus primarily on local details that have not yet been recovered from the base result.

[0307] Step S46, inference state output:

[0308] High-resolution labels are not required during inference; therefore, they are not used. and Used for loss calculation; the model executes steps S1 to S4 and outputs directly. If the system requires diagnostics or visualization, it can also retain [the necessary settings]. It serves as an auxiliary output, but does not affect the final reconstruction process.

[0309] Furthermore, in step S5, the joint optimization process during the training phase is as follows:

[0310] Step S51, pixel reconstruction loss:

[0311] Let the stitching of a real high-resolution light field sub-aperture image array be represented as: Its size and and The same HR represents the true high-resolution result, and SR represents the reconstruction result of our method. The mean absolute errors of the final reconstruction result and the basic reconstruction result are respectively:

[0312] ;

[0313] For the necessary main reconstruction supervision; To assist supervision and enable the light field feature modeling backbone to possess stable recovery capabilities, avoiding training from relying entirely on high-frequency refinement branches, the mean represents averaging over all specified elements of the tensor. This embodiment... Take 0.1, and this item is not multiplied by the auxiliary loss preheating weight.

[0314] Step S52, frequency radius-weighted view-by-view Fourier loss:

[0315] First and The images are rearranged into (BUV)×1×(sH)×(sW) per-view batches, and then orthogonally normalized two-dimensional real fast Fourier transform rfft2 is used to obtain the final reconstructed semispectral result. Semi-spectrum of the actual results Construct weights based on the normalized frequency radius:

[0316] ;

[0317] Among them, fftfreq is used to generate complete frequency axis coordinates. Generated by fftfreq, rfftfreq is used to generate the real Fourier half-spectrum frequency axis coordinates. Generated by rfftfreq The maximum radius in the current half-spectral grid is γ, and γ is the frequency emphasis exponent; in this embodiment, γ=1. The weights are then divided by their own mean to ensure a more stable loss scale across different patch sizes. The weighted absolute error is calculated for both the real and imaginary parts of the complex spectrum:

[0318] ;

[0319] and These represent the real and imaginary parts of the complex number, respectively. This loss reduces the relative weight of low-frequency components and supplements the constraints of pixel L1 on mid-to-high-frequency content.

[0320] Step S53, two-level per-view Haar wavelet loss for edge sensing:

[0321] First, use horizontal and vertical Sobel kernels. , Edge responses are generated on real high-resolution per-view images and normalized within each sample:

[0322] ;

[0323] The actual implementation will Stop the gradient and scale to the appropriate subband size using bilinear interpolation. and Perform the first-stage Haar-DWT, comparing LH, HL, and HH; then perform the second-stage Haar-DWT on the first-stage LL of both, comparing the three high-frequency subbands of the second stage. Let D = {LH, HL, HH}, its loss is:

[0324] ;

[0325] in, and These are the edge weights scaled to the two-level sub-band size, where β is the second-level relative weight (β=0.5 in this embodiment) and the edge emphasis coefficient α=0.3. The DWT in the model branch acts on the geometric fusion features to generate cues; the DWT in this loss acts on the final pixel domains SR and HR, which have different uses and input domains, and will not be confused in the text.

[0326] Step S54, pixel domain output incremental high-pass loss:

[0327] make Represent the target residual; define a high-pass operator using k×k average pooling after reflection filling, and compare the high-pass components of the predicted increment and the target residual:

[0328] ;

[0329] In this embodiment, k=3. Target residual The increment represents the unrecovered pixel difference between the true image and the base result. This comes from the refinement branch. Calculating L1 after passing through the same high-pass operator on both sides reduces the interference of low-frequency brightness differences on detail monitoring.

[0330] Step S55, epipolar plane gradient consistency and optional angle difference consistency:

[0331] The high-resolution light field sub-aperture image array is stitched together and represented as B×1×U×V× × ,in =sH、 =sW. The horizontal EPI is fixed at (v,y) and observes the (u,x) plane; the vertical EPI is fixed at (u,x) and observes the (v,y) plane. Its batch layout is as follows:

[0332] ;

[0333] For each EPI tensor, calculate the first-order difference along the last spatial axis and the penultimate angular axis, and average the L1 gradient errors of the two axes, denoted as . The horizontal and vertical EPI losses are then averaged.

[0334] ;

[0335] This feature is used to limit EPI diagonal line breakage or jitter caused by detail enhancement. This embodiment... Take 0.003. It can also calculate the co-positional difference consistency between adjacent u-viewpoints and adjacent v-viewpoints. Current stable configuration =0, therefore This is an optional step and not a necessary condition for implementing this method.

[0336] Step S56, Preheating and Joint Weighting:

[0337] Let the current training round be e, and the auxiliary loss warm-up rounds be e. ,definition:

[0338] ;

[0339] This represents the auxiliary loss weight in round e. When e=1, the preheating weight is 0, and then it increases linearly and is truncated to 1. (This is from an example.) =5. Let the auxiliary loss index set be 5. Each element corresponds to the frequency domain, wavelet, increment, EPI, and optional angle difference loss described in steps S52 to S55, respectively. The total loss is then written as:

[0340] ;

[0341] One embodiment of the parameter combination is as follows =0.1、 =0.01、 =0.005、 =0.02、 =0.003 and =0. The above values ​​are used to illustrate the achievable parameters. When a certain optional loss weight is set to 0, the corresponding term is not calculated during training.

[0342] Step S57, parameter update:

[0343] The low-resolution light field is input into the network to obtain the training output. The enabled loss term is calculated and summed according to steps S51 to S56. The parameter gradients are obtained through backpropagation, and the configured gradient optimizer is used to update the light field feature modeling backbone, orientation gating, local residual refinement branch, and shared reconstruction head. Training samples can come from light field data such as EPFL, HCInew, HCIold, INRIALytro, and Stanford Gantry. High-resolution patches are then cropped and a corresponding 4x low-resolution input is generated. For pre-processed HDF5 (Hierarchical Data Format version 5) data, a 5×5 sub-aperture image array is directly read and stitched together. Training can also be enhanced with horizontal flipping, vertical flipping, and 90-degree rotation.

[0344] Step S58, the relationship between the training and reasoning stages:

[0345] During the training phase, steps S1 to S5 are executed, where step S5 depends on... The reasoning stage is not required. Only steps S1 to S4 are executed. Therefore, the joint loss is a training method for obtaining network parameters, rather than an input condition that must be met every time spatial super-resolution is performed on the light field to be processed.

[0346] In summary, this embodiment starts with the actual sub-aperture image array stitching input, and sequentially establishes a view-by-view interpolation baseline, geometric fusion features, and basic reconstruction results. Haar-DWT is performed view-by-view on the geometric fusion features, with selective modulation of high-frequency subbands in three directions guided by LL. Then, high-frequency cue features, low-frequency structure gating, and small-scale residual injection are used to form refined features, and the final reconstruction result is obtained through a shared reconstruction head. The inputs, outputs, tensor dimensions, and training constraints of each step together form a complete and feasible technical solution, while distinguishing the replaceable optical field feature modeling backbone from the high-frequency cue and controlled residual compensation mechanism to be protected in this application.

[0347] To verify the effectiveness of the light field image spatial super-resolution method described in this invention, a 4x spatial super-resolution experiment was conducted on five publicly available light field datasets: EPFL, HCInew, HCIold, INRIA, and Stanford Gantry (STFgantry). The experimental input and output maintained a 5×5 viewpoint arrangement, with only the spatial height and width of each sub-aperture image increased to four times the input. For the original 9×9 light field, a fixed 5×5 viewpoint was used for sampling; for pre-processed data, a fixed 5×5 light field was directly read.

[0348] The training set includes 70 scenes from EPFL, 20 scenes from HCInew, 10 scenes from HCIold, 35 scenes from INRIA, and 9 scenes from STFgantry, totaling 144 scenes. The test set includes 10 scenes from EPFL, 4 scenes from HCInew, 2 scenes from HCIold, 5 scenes from INRIA, and 2 scenes from STFgantry, totaling 23 scenes. A low-resolution spatial size of 32×32 for a single sub-aperture view is used as the basic training image patch; in the 4x super-resolution task, the corresponding high-resolution image patch size is 128×128, and the preprocessing step size is 8. Data augmentation is performed during training using horizontal flipping, vertical flipping, 90-degree rotation, non-uniform angle sampling, and multi-step image patch cropping. One training implementation uses a single NVIDIA RTX 5090 graphics card with a batch size of 3.

[0349] Quantitative evaluation employs peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM), calculated on the luminance Y channel. First, the indexes are averaged across all 25 sub-aperture viewpoints within the same scene; then, the averages are calculated across all scenes within the same test set. The method of this invention is compared with EPIT, L²FMamba, and LFTransMamba, and the resulting PSNR / SSIM results are shown in Table 1.

[0350] Table 1. Quantitative comparison results of 4x spatial super-resolution (PSNR / SSIM) on five standard test sets.

[0351]

[0352] As shown in Table 1, the average PSNR and average SSIM of the method of this invention on the five test sets are 33.006 dB and 0.9483, respectively, both the highest values ​​among the listed methods. Compared with the basic method LFTransMamba, the average PSNR is improved by 0.054 dB and the average SSIM by 0.0005; compared with L²FMamba, the average PSNR is improved by 0.383 dB and the average SSIM by 0.0015; compared with EPIT, the average PSNR is improved by 0.590 dB and the average SSIM by 0.0031.

[0353] Based on the results across various datasets, the method of this invention achieves higher PSNR and SSIM simultaneously on EPFL, HCIold, and INRIA; and achieves the highest PSNR on HCInew. The experimental results do not indicate that this invention is absolutely optimal in every individual metric, but they demonstrate that the overall benefits of feature domain wavelet high-frequency cueing and controlled residual compensation are stable across different scenarios, and can improve spatial reconstruction quality while maintaining average structural similarity.

[0354] To further verify the function of each component, ablation experiments were conducted under the same training and testing conditions, and the results are shown in Table 2. Here, Baseline represents the internal baseline without the high-frequency cue branch; DWT-HF Branch represents the view-by-view feature domain wavelet high-frequency cue and controlled residual compensation branch described in this invention.

[0355] Table 2 Ablation Experiment Results

[0356]

[0357] As shown in Table 2, adding the DWT-HF branch to the baseline improved the average PSNR to 33.006 dB and the average SSIM to 0.9483, representing improvements of 0.437 dB and 0.002 respectively compared to the configuration without the DWT-HF branch. This result demonstrates that, under the same optical field feature modeling backbone, per-view feature domain Haar wavelet decomposition, LL low-frequency structure-guided high-frequency direction selection, and gated small-scale residual injection can bring measurable reconstruction gains.

Claims

1. A wavelet high-frequency cueing and residual compensation method for optical field spatial super-resolution, characterized in that, Includes the following steps: Step S1, Light field input representation and per-view bicubic interpolation baseline construction: Using the low-resolution input light field as the processing object, the light field input representation and view-by-view rearrangement are performed on the sub-aperture image array, and a view-by-view bicubic interpolation baseline is constructed. Step S2, Light Field Geometric Feature Modeling and Basic Reconstruction: Using the light field input as input, the features output by the light field feature modeling unit are fused to obtain geometric fusion features; the geometric fusion features are input to the shared reconstruction head to obtain the basic reconstruction residuals; the basic reconstruction residuals are added to the bicubic interpolation baseline for each view to obtain the basic reconstruction result. Step S3, constructing wavelet high-frequency cueing for feature domain per view: The geometric fusion features are rearranged into per-view SAI features, and a first-level two-dimensional Haar wavelet decomposition is performed on each per-view SAI feature. The low-frequency structural response is used to selectively modulate the high-frequency subbands in different directions to form high-frequency cue features. Step S4, gated local residual compensation and final reconstruction: High-frequency cue features are used to refine the local residuals of the per-view SAI features corresponding to the geometric fusion features. The detail compensation amplitude is controlled by gating and residual scaling to obtain the refined features. The final reconstruction result is obtained by adding the shared reconstruction head and the bicubic interpolation baseline for each view. Step S5, Joint optimization during training phase: During the training phase, a joint loss is constructed using the basic reconstruction results, the final reconstruction results, and the corresponding real high-resolution light field to jointly optimize steps S1 to S4.

2. The optical field spatial super-resolution method with wavelet high-frequency cueing and residual compensation according to claim 1, characterized in that, In step S1, the process of representing the light field input and constructing the bicubic interpolation baseline per viewpoint is as follows: Step S11, Sub-aperture image array stitching representation and element indexing: Let the batch size be B, the number of input luminance channels be 1, the number of viewing angles on the first and second angular axes be U and V respectively, the height and width of each low-resolution sub-aperture image be H and W respectively, and R represent the set of real numbers; the actual input received by the network is a stitched representation of the sub-aperture image array. ; The tensor size satisfies: LR indicates low resolution; Introducing batch index b, first angular axis index u, second angular axis index v, sub-aperture image internal row index y, and column index x; a six-dimensional viewpoint tensor for the low-resolution light field. ,but and Space satisfies: ； in, uH+y determines the vertical position of the y-th row in the array stitching representation within the u-th horizontal view, and vW+x determines the horizontal position of the x-th column in the array stitching representation within the v-th vertical view. Step S12, splitting by viewpoint: Will Rearranged into batches consisting of BUV individual sub-aperture images : ； in, This represents a reversible rearrangement operator that splits elements based on their perspective without changing their numerical values, but only alters their storage layout. Indicates to The results of performing a view-by-view split; each batch element corresponds to a specific (u,v) view with a size of 1×H×W; Step S13, Per-view spatial interpolation and angle arrangement restoration: First of all, Each sub-aperture image in the image is subjected to bicubic interpolation using the same spatial magnification s; Then, through The inverse rearrangement restores the original U×V angular arrangement, resulting in a bicubic interpolation baseline for each viewpoint. : ； in, This indicates an operator that performs bicubic interpolation on each sub-aperture image at a magnification of s. This represents the result of magnifying each sub-aperture image in the per-view batch by a factor of s. This represents the inverse rearrangement operator for restoring the U×V perspective arrangement. The dimensions are B×1×(UsH)×(VsW).

3. The optical field spatial super-resolution method with wavelet high-frequency cueing and residual compensation according to claim 1, characterized in that, In step S2, the process of modeling the optical field geometric features and reconstructing the basic structure is as follows: Step S21, Shallow feature mapping: First, the low-resolution light field input representation in step S1 is... Expanded along the angular dimension, it becomes a five-dimensional perspective tensor; among which... Using SAI array splicing: ； in, An angular dimension expansion operator; Indicates will In The five-dimensional tensor obtained by arranging the viewpoints into independent viewpoint dimensions; Subsequently, a 3D convolution with a kernel size of 1×3×3 was used. Map single-channel input to aisle, Indicates the number of feature channels; Next, through shallow residual mapping Extracting local features : ； in, express Shallow mapping features after initial 1×3×3 3D convolution; This indicates that the shallow-mapped features are mapped through shallow residual mapping. The obtained local residual characteristics; and All have the same dimensions and are added element by element; The shallow residual mapping It consists of three consecutive combined layers, each of which includes a 1×3×3 3D convolution and LeakyReLU; During training, local features Rearranged by perspective as (BUV) × (HW) × The token sequence is randomly masked by removing 35% of the tokens, and the original layout is restored; random masking is not performed during inference. Step S22, Light field feature modeling and backbone feature extraction: The backbone is modeled using light field features consistent with the output interface size; in local features Add learnable angle embeddings to The rearrangement is (BUV)×H×W× The system connects 12 light field feature modeling units in series; the 12 light field feature modeling units are divided into 4 groups; each group contains three units, namely SAI modeling unit, macro-pixel modeling unit and EPI modeling unit; the light field feature modeling units are modeled using convolution, attention or state space sequence. The recursive process for modeling the light field features is expressed as follows: ； in, Initial characteristics of the skeletal structure; Indicates from B× ×(UV)×H×W to (BUV)×H×W× Layout transformation, For learnable angle embedding, size and same; Output for the nth light field feature modeling unit; This represents the mapping of the nth light field feature modeling unit; It is The result after switching to a per-view batch layout; Step S23, Multi-level feature aggregation and shallow bypass fusion: First, after each light field feature modeling unit is executed, the outputs of the 3rd, 6th, 9th, and 12th units are saved respectively. , , and ; Then, output the four groups of data. , , and A multi-level aggregation mapping is constructed by splicing along the channel dimension and performing 3×3 two-dimensional convolution and LeakyReLU. Compress back aisle; Subsequently, it was restored to B× The SAI array splicing features of ×(UH)×(VW) and with The restored shallow bypass features are added element by element to obtain several what Fusion features : ； in, Indicates channel-dimensional splicing. This indicates the restoration from the per-view feature layout to the SAI array stitched feature layout; This indicates that shallow features The SAI array splicing features are restored to B×C_f×(U·H)×(V·W); Step S24, Reconstructing the Basic Results: Shared Reconstruction Head The sequence includes a 1×1 2D convolution, a PixelShuffle with a scaling factor of s, LeakyReLU, and a 3×3 2D convolution; the first convolution will... Channel mapping as The s² channel pixel rearrangement upsampling operator rearranges the channel information into the spatial dimension, finally outputting the single-channel reconstruction residual; the single-channel reconstruction residual is then compared with the per-view bicubic interpolation baseline. The summation yields the basic reconstruction result. : ； in, Indicates geometric fusion features Input shared reconstruction head The resulting single-channel foundation reconstruction residual has dimensions similar to same.

4. The optical field spatial super-resolution method with wavelet high-frequency cueing and residual compensation according to claim 1, characterized in that, In step S3, the process of constructing the per-view feature domain wavelet high-frequency cue is as follows: Step S31, Geometric fusion feature per-view rearrangement and boundary processing: Geometric fusion features Decomposed into BUV per-view SAI features : ； Each view-by-view SAI feature contains Each channel shares the same set of Haar transform and direction gating parameters; if H or W is odd, then each channel is calculated separately. and , and These represent the fill amounts in the height and width directions, respectively, and are only added on the right side. Column, supplement below Okay, use reflective fill. For reflection filling operators, This indicates the view-by-view features after reflection filling: ； When both H and W are even numbers = =0, no padding is performed; Step S32, First-order per-view 2D Haar-DWT: Step S321, odd / even position sub-sampling: After filling Sampling is performed in both row and column directions according to odd and even positions, resulting in four sets of tensors: ； ； in, Corresponding to even rows and even columns, Corresponding to even rows and odd columns, Corresponding to odd-numbered rows and even-numbered columns, The four tensors have the same size; i and j represent the row and column indices in the Haar subband, respectively, and are both integers starting from 0. Step S322: Calculate the four Haar subbands using an orthogonal two-dimensional Haar transform coefficient of 0.

5. ； ； ； ； Wherein, LL is the low-frequency subband; LH is the column-direction differential high-frequency subband; HL is the row-direction differential high-frequency subband; and HH is the row-column joint differential high-frequency subband. Step S323, Dimension Recording: The spatial dimensions of the four Haar subbands are all Simultaneously record the original dimensions (H, W) before wavelet transform; Step S33, three-way high-frequency gating guided by low-frequency structure: Step S331, Gated Tensor Generation: Input LL into a lightweight gated network consisting of 1×1 convolutions, GELU, 1×1 convolutions, and a Sigmoid function; the first convolution holds... Channel, second convolution output aisle, Represents the Sigmoid function, gated tensor It is expressed as follows: ； in, This indicates that both the number of input and output channels are 1. The first 1×1 two-dimensional convolution; Indicates will Channel mapping is 3 The second 1×1 two-dimensional convolution of the channel; Step S332, Directional Splitting: Along the channel Divide into three groups on average: ,and The three sets of gating tensors correspond one-to-one with LH, HL, and HH, respectively, and all have the shape of LH, HL, and HH. ; Step S333, directional modulation centered on the identity mapping: set up This represents the original high-frequency subband in direction d. For directional gating, a learnable scalar is used. The following operations are performed to obtain the modulated subband. : ； in, For the original high-frequency subband in direction d, For corresponding gating, For the modulated subband, ⊙ represents element-wise multiplication, and the parameter... Initialize to 0; Step S34, reconstruction of high-frequency cue features and low-frequency structural features: Construct a zero tensor with the same shape as LL: Set the low-frequency input to zero, perform an inverse Haar transform on the three modulated high-frequency subbands, and clip them back to H×W to obtain the high-frequency cue features. : ； in, This indicates that the right and bottom fill generated in step S31 will be trimmed. This represents the two-dimensional Haar inverse wavelet transform; simultaneously, the LL subband is used as the only non-zero subband to perform the inverse transform and is then pruned to obtain the low-frequency structural features. : ； in, and The dimensions are all the same as same, , and All three participate in residual compensation without changing the number of viewpoints or spatial layout.

5. The optical field spatial super-resolution method with wavelet high-frequency cueing and residual compensation according to claim 1, characterized in that, In step S4, the gated local residual compensation and final reconstruction process is as follows: Step S41, High-frequency cue feature-driven local residual prediction: First, the per-view SAI features obtained in step S31 are... High-frequency cue features obtained in step S34 By splicing along the channel dimension, the fused feature Q is obtained: ； Step S411, Channel Compression: Use 1×1 convolution Put Q's Channel compression to Channels for fusing per-view SAI features and high-frequency prompt features ; Step S412, candidate residual output: The features compressed in step S411 are sequentially input into four cascaded lightweight local convolution residual blocks for local feature transformation, and then subjected to a 1×1 two-dimensional convolution at the end. Obtaining and per-view SAI features Candidate residuals of the same size : ； in, This represents four lightweight local convolutional residual blocks. The first sub-block of each block sequentially performs channel-dimensional layer normalization, 1×1 channel expansion, 3×3 depthwise convolution, channel binary multiplication gate, and 1×1 channel recovery, and then performs residual connections with zero initialization. The second sub-block sequentially performs LayerNorm, 1×1 expansion, SimpleGate, and 1×1 recovery, and then performs residual connections with zero initialization. Step S42, generation of detail gating and low-frequency structure gating: Two sets of independent 1×1 convolutions are used to generate detail gating. and low-frequency structure gating : ； in, and These represent two independent 1×1 two-dimensional convolutions. Indicates the Sigmoid function; Step S43, joint gating and small-scale residual injection: First, use learnable scalars Low-frequency structure gating Convert to a 1-centered adjustment factor and multiply it element-wise with the detail gating: ； in, Indicates joint gating, This is a low-frequency structure gated scaling scalar, initialized to 0; Then, learnable small-scale residual proportions were used. Controlling the overall magnitude of candidate residuals: ； in, This represents the scaling factor for the overall residuals. Step S44, refine feature recovery and shared reconstruction: Will Restore the SAI array stitching features according to the original U×V viewpoint order to obtain the array stitching features. : ； in, The dimensions are B× ×(UH)×(VW); Will Input to the same shared reconstruction head described in step S24 and the same as obtained in step S13 Add them together to get the final reconstruction result. : ； in, To ultimately reconstruct the residuals; Step S45, Construction of pixel domain output increment in training state: Output during training and And construct pixel-domain detail increment : ； Where stopgrad represents an operator that keeps the value unchanged but blocks gradient backpropagation; Step S46, inference state output: High-resolution labels are not required during inference, and the underlying reconstruction results are not used. and pixel domain detail increment Used for loss calculation; during the inference phase, steps S1 to S4 are executed sequentially and the final reconstruction result is output. If the system requires diagnostics or visualization, then retain [this option]. As an auxiliary output.

6. The optical field spatial super-resolution method with wavelet high-frequency cueing and residual compensation according to claim 1, characterized in that, In step S5, the joint optimization process during the training phase is as follows: Step S51, pixel reconstruction loss: Let the stitching of a real high-resolution light field sub-aperture image array be represented as: Size and and The same applies, where HR represents high resolution and SR represents super-resolution. Represents a true high-resolution light field. This indicates the final super-resolution reconstruction result, which is the final reconstruction result. and The mean absolute errors are as follows: ； in, For necessary main reconstruction supervision; To assist in supervision; mean indicates averaging over all specified elements of the tensor; Step S52, frequency radius-weighted view-by-view Fourier loss: First and The images are rearranged into (BUV)×1×(sH)×(sW) per-view batches, and then orthogonally normalized two-dimensional real fast Fourier transform rfft2 is used to obtain the final reconstructed semispectral result. Semi-spectrum of the actual results Construct weights based on the normalized frequency radius: ； in, Represents the normalized frequency radius. Indicates the frequency radius weight. These represent the vertical and horizontal frequency coordinates, respectively. `fftfreq` is used to generate the complete frequency axis coordinates. This represents the maximum frequency radius of the half-spectrum grid. Indicates frequency emphasis index, Generated by fftfreq, rfftfreq is used to generate the real Fourier half-spectrum frequency axis coordinates. Generated by rfftfreq; the weighted absolute error is calculated for the real and imaginary parts of the complex spectrum, respectively: ； in, and represents the real part and imaginary part of a complex number, respectively, and fft represents the Fast Fourier Transform; Step S53, two-level view-by-view Haar wavelet loss for edge sensing: First, use horizontal and vertical Sobel kernels. , Edge responses are generated on real high-resolution per-view images and normalized within each sample: ； in, Represents the edge response of a real image; Indicates the normalized edge weights; Indicates the edge emphasis coefficient; for and Perform the first level of Haar-DWT, comparing LH, HL, and HH; then perform separate operations on... and The first-stage LL continues with the second-stage Haar-DWT, comparing the three high-frequency subbands of the second stage; let the set of directional high-frequency subbands D = {LH, HL, HH}, and the loss... for: ； in, Index for the direction subband; express In the direction of level 1 High-frequency subband; express In the direction of level 2 High-frequency subband; express The corresponding subband; express The corresponding subband; and These are the edge weights scaled to the two-level sub-band size, respectively; β is the second-level relative weight. Step S54, pixel domain detail increment Qualcomm loss: Make the target residual Define a high-pass operator that uses k×k average pooling after reflection filling, and compare the high-pass components of the predicted increment and the target residual: ； in, Represents any input tensor of a high-pass operator. Indicates reflection filling. This represents k×k average pooling. Indicates the corresponding high-pass operator; Indicates pixel-level detail increment; Target residual The increment represents the unrecovered pixel difference between the true image and the base result. L1 is calculated from the refinement branch; both sides pass through the same high-pass operator before L1 is calculated. Step S55, epipolar plane gradient consistency and angle difference consistency: The high-resolution light field sub-aperture image array is stitched together and represented as B×1×U×V× × ,in =sH、 =sW; Fix the horizontal EPI to (v,y) and observe the (u,x) plane, fix the vertical EPI to (u,x) and observe the (v,y) plane, batch layout. for: ； in, Represents high-resolution light field data; For horizontal EPI rearrangement operators; For the vertical EPI rearrangement operator, calculate the first-order difference for each EPI tensor along the last spatial axis and the penultimate angular axis, and average the L1 gradient errors of the two axes, denoted as . The horizontal and vertical EPI losses are then averaged to obtain the epipolar plane gradient consistency loss L. epi : ； represent the horizontal EPI tensors obtained from the final reconstruction result and the true high-resolution light field through horizontal EPI rearrangement, respectively; These represent the vertical EPI tensors obtained from the final reconstruction result and the true high-resolution light field through vertical EPI rearrangement, respectively. Step S56, Preheating and Joint Weighting: Let the current training round be e, and the auxiliary loss warm-up rounds be e. ,definition: ； in, Let e ​​represent the auxiliary loss weight in round e. When e=1, the warm-up weight is 0, then it increases linearly and is truncated to 1; let the auxiliary loss index set be... Each element corresponds to the frequency domain, wavelet, increment, EPI, and optional angle difference loss described in steps S52 to S55, respectively. The total loss is then calculated as follows: Represented as: ； in, Indicates the auxiliary loss category, This indicates the weight corresponding to the auxiliary loss category; Step S57, parameter update: Inputting low-resolution light field representation Enter steps S1 to S4 sequentially to obtain... , and The enabled loss term is calculated and summed according to steps S51 to S56. The parameter gradient is obtained through backpropagation. Then, the configured gradient optimizer is used to update the light field feature modeling backbone, orientation gating, local residual refinement branch and shared reconstruction head. Step S58, the relationship between the training and reasoning stages: Training phase and As input, steps S1 to S5 are executed to determine each learnable parameter; the parameters are fixed during the inference phase, using only... Execute steps S1 to S4 as input and output. .

7. An electronic device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images