Light field image super-resolution processing method and system

Abstract

The embodiment of the invention provides a light field image super-resolution processing method. The method is applied to the technical field of computer vision and image processing and comprises the steps of obtaining a degraded light field image, obtaining an original light field image and obtaining a multi-view sub-aperture image; extracting and aligning the multi-view-angle sub-aperture images, extracting SAI sequences of each view angle through a light field decoding tool, and aligning all the SAI sequences to a central view angle coordinate system through an optical flow method or a geometric transformation method; cross-view feature extraction is carried out on the SAI sequence, restoration is completed by calculating attention weight and carrying out denoising processing, and the restored SAI sequence is optimized; performing depth estimation on the optimized SAI sequence, generating a disparity map, and performing depth map conversion to obtain a depth image; and carrying out super-resolution processing, edge sharpening and texture generation. In this way, the problems of view angle inconsistency, detail loss and the like in the prior art are effectively solved.

Description

Technical Field

[0001] The present disclosure relates to the technical fields of computer vision and image processing, and in particular to a light field image super-resolution processing method and system. Background Art

[0002] Light field images are different from ordinary images. They record the direction and position of light in space and are usually captured by microlens arrays or multi-camera arrays. These images are widely used in fields such as depth estimation, refocusing, and virtual reality. Traditional image restoration and super-resolution techniques are usually processed step by step and independently, resulting in information loss and low efficiency. The multi-perspective characteristics of light field images have not been fully explored, and consistency between perspectives is difficult to ensure during the restoration process. The light field depth information is not utilized in super-resolution, resulting in blurred textures and distorted edges. It is necessary to combine restoration and super-resolution to optimize the system by utilizing the multi-perspective and depth information of light field data.

[0003] Therefore, there is an urgent need for a design solution that can effectively perform super-resolution processing of light field images. Summary of the Invention

[0004] The present disclosure provides a light field image super-resolution processing method and system, which achieves image super-resolution processing by performing motion estimation on light field sub-aperture images and converting disparity maps and depth maps, and at least solves the technical problems of information loss and low efficiency in existing image restoration and super-resolution technologies.

[0005] According to a first aspect of the present disclosure, a light field image super-resolution processing method is provided, comprising the following steps: Obtain a degraded light field image, obtain the original light field image through a photographic acquisition device, and obtain a multi-view sub-aperture image; Extracting and aligning the multi-view sub-aperture images, extracting the SAI sequences of each view using a light field decoding tool, and aligning all SAI sequences to a central view coordinate system using an optical flow method or a geometric transformation method; Performing cross-view feature extraction on the SAI sequence, repairing the sequence by calculating attention weights and performing denoising processing, and optimizing the repaired SAI sequence to obtain an optimized SAI sequence; Perform depth estimation on the optimized SAI sequence, generate a disparity map and perform depth map conversion to obtain a depth image; The depth image is subjected to super-resolution processing, and edge sharpening and texture generation are performed.

[0006] According to the above aspects and any possible implementation, an implementation is further provided, wherein cross-view feature extraction is performed on the SAI sequence, restoration is completed by calculating attention weights and performing denoising, and the process of optimizing the restored SAI sequence is as follows: A convolutional neural network is used for cross-view feature extraction to extract the feature map of each SAI, which is then fed into the Transformer module to calculate the attention weights between views. Based on the attention weights, weighted aggregation of valid pixels from adjacent perspectives is performed to fill the occluded area of the current perspective; Perform adaptive weighted filtering and denoising on each SAI block of the filled sub-aperture image to perform repair; A generative adversarial network is used to perform drive enhancement optimization and multi-view consistency optimization on the repaired SAI sequence.

[0007] According to the above aspects and any possible implementation, an implementation is further provided, wherein the process of performing adaptive weighted filtering denoising on each padded sub-aperture image SAI block to perform repair is as follows: Calculate the standard deviation of each sub-aperture image SAI block and use it as the noise intensity indicator; The filtering weight of each pixel position of the SAI is calculated based on the noise intensity index, and the multi-core results are fused according to the filtering weight to complete the restoration.

[0008] According to the above aspects and any possible implementation, an implementation is further provided, wherein the process of using a generative adversarial network to perform drive enhancement optimization and multi-view consistency optimization on the repaired SAI sequence is as follows: Input the restored SAI image into the generator, output a detail-enhanced image, and use a discriminator to determine the distribution difference between the enhanced image and the true clear image, and perform image correction based on the difference; Using a pre-trained VGG-19 network and combining it with a light field depth map to process the detail-enhanced image, thereby obtaining an enhanced image that is semantically consistent with the real image and has consistent geometric structure and depth information; Using L1 loss and perceptual loss to train the generator, and restore details of the enhanced image; The optical flow method is used to calculate the multi-view motion vectors of the image after detail restoration, and forced restoration is performed. The depth consistency constraint is then used to optimize the restored image to complete the optimization.

[0009] According to the above aspects and any possible implementation, an implementation is further provided, wherein the process of performing depth estimation on the optimized SAI sequence, generating a disparity map, and performing depth map conversion is as follows: Input the optimized SAI sequence, select the central view as the reference view and the adjacent view as the target view, use the improved MC-CNN network to extract the multi-scale features of the SAI sequence through the convolution layer, obtain the feature map, and splice it to generate the disparity map; For each pixel of the disparity map, calculating the similarity between the reference perspective and the target perspective within a disparity search range, and performing dynamic optimization; Filling holes in the disparity map using an effective disparity median or mean filling method, and smoothing the disparity result using a Markov random field to eliminate noise in the disparity map; The disparity map is converted into a depth map according to camera intrinsic parameters.

[0010] According to the above aspects and any possible implementation, a further implementation is provided, wherein the method for filling holes in the disparity map using the effective disparity median or mean filling method is: ;in, is the disparity value output after filtering, is the spatial domain standard deviation, is the color domain standard deviation, W p is the normalization coefficient, p and q are pixel coordinates, ||pq|| is the spatial Euclidean distance between pixel p(x,y) and neighboring pixel q(i,j), ||I(p)-I(q)|| is the brightness difference between pixels p and q in the original light field image, D(q) is the original disparity value or depth value of the neighboring pixel q(i,j), and Ω is the neighborhood window of the current pixel p(x,y).

[0011] According to the above aspects and any possible implementation, an implementation is further provided, wherein the process of performing super-resolution processing on the depth image and performing edge sharpening and texture generation is as follows: Performing non-uniform upsampling on the depth map, and extracting features using a multi-scale feature fusion network to achieve depth map fusion and feature enhancement; A gradient difference term is introduced into the loss function, and based on the gradient difference term, the edge alignment between the super-resolved image and the high-resolution real image is forced to be completed to achieve edge sharpening; The sharpened depth map is spliced with the low-resolution image and input into the generator to output a high-resolution image.

[0012] According to the above aspects and any possible implementation, an implementation is further provided, wherein the process of performing non-uniform upsampling on the depth map and extracting features using a multi-scale feature fusion network to achieve depth map fusion and feature enhancement is as follows: Divide the depth map into foreground, midground, and background, upsample the foreground area by a factor of 2, upsample the midground area by a factor of 1.5, and upsample the background area by a factor of 1 or 1.2; The low-level and high-level features of the low-resolution image are extracted through the residual network, and the depth map is input as an additional channel. It is spliced with the low-level and high-level features or weightedly fused through the attention mechanism to complete the depth map fusion and feature enhancement.

[0013] According to a second aspect of the present disclosure, there is provided a light field image super-resolution processing system, comprising: a light field data acquisition module, a light field data parsing module, a multi-view restoration module, a depth estimation module, and a super-resolution generation module; The light field data acquisition module is used to acquire degraded light field images, obtain original light field images through a photographic acquisition device, and obtain multi-view sub-aperture images; The light field data parsing module is used to extract and align the multi-view sub-aperture images, extract the SAI sequences of each view through the light field decoding tool, and align all SAI sequences to the central view coordinate system through the optical flow method or the geometric transformation method; The multi-view restoration module is used to extract cross-view features of the SAI sequence, complete the restoration by calculating the attention weight and performing denoising processing, and optimize the restored SAI sequence to obtain an optimized SAI sequence; The depth estimation module is used to perform depth estimation on the optimized SAI sequence, generate a disparity map and perform depth map conversion to obtain a depth image; The super-resolution generation module is used to perform super-resolution processing on the depth image, and perform edge sharpening and texture generation.

[0014] Compared with the prior art, the present invention has the following technical effects: The present invention discloses an integrated processing method and system for image restoration and super-resolution enhancement. By capturing light field images, aligning multi-view features, and collaboratively designing a restoration method based on a generative adversarial network and a super-resolution module, the method solves problems such as inconsistent viewpoints and loss of details in the prior art. While maintaining low computational complexity, the method significantly improves the reconstruction quality of light field images and can be applied to fields such as VR / AR and medical microscopy.

[0015] It should be understood that the contents described in the Summary of the Invention section are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which: Figure 1 A schematic flow chart of a light field image super-resolution processing method according to an embodiment of the present disclosure is shown; Figure 2 A schematic diagram of a multi-view attention restoration network structure of a light field image super-resolution processing method according to an embodiment of the present disclosure is shown; Figure 3 A schematic diagram of a depth-guided upsampling process of a light field image super-resolution processing method according to an embodiment of the present disclosure is shown; Figure 4 A schematic structural diagram of a light field image super-resolution processing system according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0017] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Reference Figure 1 As shown, this embodiment provides a light field image super-resolution processing method, comprising the following steps: S101 , obtaining a degraded light field image, obtaining an original light field image through a photographic acquisition device, and obtaining a multi-view sub-aperture image.

[0020] S102: Extract and align the multi-view sub-aperture images, extract the SAI sequences of each view using a light field decoding tool, and align all SAI sequences to a central view coordinate system using an optical flow method or a geometric transformation method.

[0021] S103 , extracting cross-view features from the SAI sequence, repairing the sequence by calculating attention weights and performing denoising, and optimizing the repaired SAI sequence.

[0022] like Figure 2As shown, in this embodiment, a convolutional neural network (CNN) is used to perform cross-view feature extraction, and a convolutional neural network (CNN) is used to extract the feature map of each SAI, and the multi-view feature map is input into the Transformer module to calculate the attention weights between the views.

[0023] Then, this embodiment uses the self-attention mechanism to calculate the similarity weight w between the pixels in the current view occluded area and the pixels in the adjacent view non-occluded area. ij , the formula is: (1) Among them, Q i is the current view feature, K j is the adjacent view feature, and d is the feature dimension.

[0024] After calculating the similarity weight, this embodiment performs pixel fusion and filling. According to the attention weight, the effective pixels of adjacent perspectives are weighted and aggregated to fill the occluded area of the current perspective. Specifically: (2) Among them, I j (x,y) is the pixel value of the adjacent view.

[0025] For each sub-aperture image SAI block, this embodiment performs adaptive weighted filtering denoising, and calculates its standard deviation ( ) as an indicator of noise intensity: (3) in, is the mean value of pixels in the block, and N is the total number of pixels.

[0026] Differentiate between Gaussian noise, salt and pepper noise, etc. by using noise histogram or frequency domain analysis (such as Fourier transform). Use different filter kernel sizes and weight functions in different areas. Large kernels smooth noise, small kernels retain details, and avoid excessive blurring through weighted fusion. Calculate the filter weight for each pixel position (i, j) and fuse the multi-kernel results I filtered (i,j): (4) in, is the weight.

[0027] Weight By noise intensity And the edge strength E(i,j) jointly determine: (5) in, are the weight ratio parameters for controlling noise and edge respectively.

[0028] E(i,j) calculates the gradient magnitude through the Soble operator: (6) Finally, dynamic search range and adaptive similarity threshold are introduced to improve the NLM formula: (7) Among them, the dynamic search window , the smooth area (large window) increases the number of similar blocks, and the edge area (small window) reduces cross-edge mismatching. w(p,q) is the adaptive similarity weight, specifically: (8) Among them, h is the smoothing parameter, according to the local Dynamic Adjustment: (9) in, is the attenuation coefficient, such as 0.1, h base is the basic smoothing parameter, which is a predefined initial smoothing strength (designed for moderate noise levels).

[0029] Subsequently, this embodiment uses L1 loss and perceptual loss to train the generator, restores details of the enhanced image, and uses the optical flow method to calculate the multi-view motion vector of the image after detail restoration, performs forced restoration, and constructs a depth consistency constraint optimized restoration image to complete the restoration. The specific process is as follows: The generator adopts a U-Net structure, inputs the repaired SAI, and outputs a detail-enhanced image. The discriminator judges the distribution difference between the generated image and the real clear image, guiding the generator to restore the real texture.

[0030] Wasserstein GAN (WGAN-GP) is used to improve training stability. Gradient penalty term: (10) For perceptual loss, we use the pre-trained VGG-19 network to extract features and force the generated images to be consistent with the real images at the semantic level: (11) Among them, L perc is the perceptual loss of the VGG network feature layer, is the VGG layer l feature, G(z) is the predicted depth map of the super-resolution image, ( X real ) is the input depth map.

[0031] Combined with the light field depth map, ensure that the geometric structure of the generated image is consistent with the depth information: (12) Where: Ldepth is the depth estimation loss, G(z) is the predicted depth map of the super-resolution image, Depth(x real ) is the input depth map.

[0032] This example uses only L1 loss and perceptual loss to train the generator, preliminarily restoring details. The generator is fixed, and the discriminator is trained to distinguish generated images from real images. The generator and discriminator are optimized alternately.

[0033] Furthermore, the specific steps for multi-view consistency optimization are as follows: Optical flow constraint, which uses the optical flow method to calculate the motion vector between multi-view images, forcing the restoration results to be consistent across views. The formula is: (13) Where: L flow is the optical flow consistency loss, which is used to measure the difference in motion vectors between the restored image and the real image in multiple perspectives. V is the total number of perspectives involved in the calculation. I repair is the restored sub-aperture image (SAI), is a true non-degraded light field image, Flow(A,B) is the optical flow field from image A to image B, and represents the pixel-level motion vector (Δx, Δy).

[0034] Depth consistency constraint uses the light field depth map to ensure that the geometric structure of the restored image is reasonable. The formula is: (14) Where: L depth is the depth estimation loss, Depth(I repair ) is the repaired depth map, is the input depth map.

[0035] S104: perform depth estimation on the repaired SAI sequence, generate a disparity map, and perform depth map conversion.

[0036] In this embodiment, the specific steps of performing depth estimation on the light field image and generating the disparity map are as follows: The system takes a sequence of restored light field sub-aperture images (SAI) as input, selects the central view as the reference view, and the adjacent views as the target view. Using an improved MC-CNN network structure, it extracts multi-scale features through convolutional layers and concatenates the feature maps from multiple adjacent views to enhance the robustness of matching in occluded areas.

[0037] For each pixel, the similarity between the reference view and the target view is calculated within the disparity search range (e.g., 0-64 pixels).

[0038] (15) Among them, p is the pixel coordinate, d is the candidate disparity value, and cosine similarity is used to measure feature matching. Feature(p ref ) is the feature of the reference perspective, Feature(p target +d) is the characteristic of the target perspective.

[0039] Optimize the cost volume through dynamic programming to minimize the energy function: (16) Among them, C(p,D p ) is the matching cost, which penalizes the disparity mutation of adjacent pixels p,q. is the weight, and T is the cutoff threshold.

[0040] Fill the holes in the disparity map using the median or mean of the surrounding valid disparities. Based on the edge information of the original light field image, perform edge-preserving smoothing on the disparity map: (17) in, is the disparity value output after filtering, is the spatial domain standard deviation, is the color domain standard deviation, W p is the normalization coefficient, p and q are pixel coordinates, ||pq|| is the spatial Euclidean distance between pixel p(x,y) and its neighboring pixel q(i,j), ||I(p)-I(q)|| is the brightness difference between pixels p and q in the original light field image, D(q) is the original disparity value or depth value of the neighboring pixel q(i,j)q(i,j), and Ω is the neighborhood window of the current pixel p(x,y).

[0041] To optimize the light field perspective consistency, Markov random field (MRF) is used to smooth the disparity results and eliminate noise. Combining the disparity smoothing constraint with the multi-view optical flow consistency: (18) in, is a unary term based on the matching cost and depth prediction confidence, It is a binary term that forces the disparity of adjacent pixels to be continuous and consistent with the optical flow field motion estimation.

[0042] The energy function is minimized through iterative optimization to eliminate local inconsistencies in the disparity map.

[0043] Furthermore, the specific steps of converting the disparity map into the depth map are: The disparity map is converted into a depth map based on the camera's intrinsic parameters (baseline distance, focal length). The depth value is mapped to the range [0,1] as the prior input of the super-resolution module.

[0044] (19) Where Z(x,y) is the depth value, f is the focal length of the camera (in pixels, which must be obtained through calibration), B is the baseline distance (physical length, unit: meter) between adjacent viewpoints, and D(x,y) is the disparity value of the pixel point (x,y) (unit: pixel).

[0045] Map the depth value to the range [0,1] for easy subsequent processing: (20) Among them, Z min is the minimum depth value, Z max is the maximum depth value.

[0046] S105 , performing super-resolution processing on the depth image, and performing edge sharpening and texture generation.

[0047] like Figure 3 As shown, in this embodiment, super-resolution is performed on the depth image. First, the depth map is non-uniformly up-sampled, and a multi-scale feature fusion network is used to perform feature extraction, depth map fusion, and feature enhancement. Then, edge sharpening and texture generation are performed.

[0048] Perform non-uniform upsampling on the depth map. The specific steps are: Based on the depth map, the image is divided into foreground (depth value < 0.3), midground (0.3 ≤ depth value ≤ 0.7), and background (depth value > 0.7). The foreground region is upsampled by a factor of 2, and deconvolution is used to enhance detail. The midground region is upsampled by a factor of 1.5 to balance detail and computational efficiency. The background region is upsampled by a factor of 1 or a light factor of 1.2 to avoid excessive noise amplification. A small convolution kernel (e.g., 3×3) is used for the foreground to capture local texture. A medium convolution kernel (e.g., 5×5) is used for the midground to balance detail and smoothness. A large convolution kernel (e.g., 7×7) is used for the background to smooth noise.

[0049] Use multi-scale feature fusion network for feature extraction, depth map fusion and feature enhancement. The specific steps are: A residual network (ResNet) extracts low-level (edge) and high-level (semantic) features from low-resolution images. Low-level features (edges and textures) are extracted through shallow convolutional layers. High-level features (semantic information) are extracted through residual blocks and densely connected blocks. The depth map is fed into a separate branch to generate a depth-aware mask to guide feature selection. Based on the edge information in the depth map, details in the foreground are enhanced. Irrelevant feature channels in the background are suppressed to reduce computational redundancy. Pyramid pooling is combined to capture multi-scale contextual information.

[0050] The depth map is input as an additional channel and concatenated with the image features or weightedly fused through the attention mechanism.

[0051] Use dilated convolution to expand the receptive field and capture long-distance dependencies.

[0052] The specific steps of edge sharpening and texture generation are: Introducing the gradient difference term into the loss function forces the super-resolved image to align with the edges of the high-resolution true image: (twenty one) in, The gradient map calculated by the Sobel operator, I SR is the super-resolution image, I HR For real images.

[0053] GAN drives texture synthesis. The generator outputs a high-resolution image, and the input is a concatenation of a low-resolution image and a depth map. The discriminator determines whether the generated image is realistic and evaluates its spatial consistency with the depth map. The resulting adversarial loss is: (twenty two) Where: L GAN is the adversarial loss, D is the discriminator, which outputs the probability of image authenticity, G is the generator, which inputs the restored image and generates an image with enhanced details, I real is a true non-degraded image, I repair is the restored image; like Figure 4 As shown, this embodiment also provides a light field image super-resolution processing system, including: a light field data acquisition module 1, a light field data analysis module 2, a multi-view restoration module 3, a depth estimation module 4 and a super-resolution generation module 5; The light field data acquisition module 1 is used to obtain degraded light field images, obtain original light field images through a photographic acquisition device, and obtain multi-view sub-aperture images; The light field data parsing module 2 is used to extract and align the multi-view sub-aperture images, extract the SAI sequences of each view through the light field decoding tool, and align all SAI sequences to the central view coordinate system through the optical flow method or geometric transformation method; The multi-view restoration module 3 is used to extract cross-view features of the SAI sequence, complete the restoration by calculating the attention weight and performing denoising processing, and optimize the restored SAI sequence to obtain an optimized SAI sequence; The depth estimation module 4 is used to perform depth estimation on the optimized SAI sequence, generate a disparity map and perform depth map conversion to obtain a depth image; The super-resolution generation module 5 is used to perform super-resolution processing on the depth image, and perform edge sharpening and texture generation.

[0054] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0055] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0056] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A light field image super-resolution processing method, characterized in that: The following steps are involved: Obtain a degraded light field image, obtain the original light field image through a photographic acquisition device, and obtain a multi-view sub-aperture image; Extracting and aligning the multi-view sub-aperture images, extracting the SAI sequences of each view using a light field decoding tool, and aligning all SAI sequences to a central view coordinate system using an optical flow method or a geometric transformation method; Performing cross-view feature extraction on the SAI sequence, repairing the sequence by calculating attention weights and performing denoising processing, and optimizing the repaired SAI sequence to obtain an optimized SAI sequence; Perform depth estimation on the optimized SAI sequence, generate a disparity map and perform depth map conversion to obtain a depth image; The depth image is subjected to super-resolution processing, and edge sharpening and texture generation are performed.

2. The light field image super-resolution processing method according to claim 1, characterized in that: The process of extracting cross-view features of the SAI sequence, repairing the sequence by calculating attention weights and performing denoising, and optimizing the repaired SAI sequence is as follows: A convolutional neural network is used for cross-view feature extraction to extract the feature map of each SAI, which is then fed into the Transformer module to calculate the attention weights between views. Based on the attention weights, weighted aggregation of valid pixels from adjacent perspectives is performed to fill the occluded area of the current perspective; Perform adaptive weighted filtering and denoising on each SAI block of the filled sub-aperture image to perform repair; A generative adversarial network is used to perform drive enhancement optimization and multi-view consistency optimization on the repaired SAI sequence.

3. The light field image super-resolution processing method according to claim 2, characterized in that: The process of performing adaptive weighted filtering denoising on each SAI block of the padded sub-aperture image is as follows: Calculate the standard deviation of each sub-aperture image SAI block and use it as the noise intensity indicator; The filtering weight of each pixel position of the SAI is calculated based on the noise intensity index, and the multi-core results are fused according to the filtering weight to complete the restoration.

4. The light field image super-resolution processing method according to claim 2, characterized in that: The process of using the generative adversarial network to perform drive enhancement optimization and multi-view consistency optimization on the repaired SAI sequence is as follows: Input the restored SAI image into the generator, output a detail-enhanced image, and use a discriminator to determine the distribution difference between the enhanced image and the true clear image, and perform image correction based on the difference; Using a pre-trained VGG-19 network and combining it with a light field depth map to process the detail-enhanced image, thereby obtaining an enhanced image that is semantically consistent with the real image and has consistent geometric structure and depth information; Using L1 loss and perceptual loss to train the generator, and restore details of the enhanced image; The optical flow method is used to calculate the multi-view motion vectors of the image after detail restoration, and forced restoration is performed. The depth consistency constraint is then used to optimize the restored image to complete the optimization.

5. The light field image super-resolution processing method according to claim 1, characterized in that: The process of performing depth estimation on the optimized SAI sequence, generating a disparity map and performing depth map conversion is as follows: Input the optimized SAI sequence, select the central view as the reference view and the adjacent view as the target view, use the improved MC-CNN network to extract the multi-scale features of the SAI sequence through the convolution layer, obtain the feature map, and splice it to generate the disparity map; For each pixel of the disparity map, calculating the similarity between the reference perspective and the target perspective within a disparity search range, and performing dynamic optimization; Filling holes in the disparity map using an effective disparity median or mean filling method, and smoothing the disparity result using a Markov random field to eliminate noise in the disparity map; The disparity map is converted into a depth map according to camera intrinsic parameters.

6. The light field image super-resolution processing method according to claim 5, characterized in that: The method of filling holes in the disparity map using the effective disparity median or mean filling method is: ;in, is the disparity value output after filtering, is the spatial domain standard deviation, is the color domain standard deviation, W p is the normalization coefficient, p and q are pixel coordinates, ||pq|| is the spatial Euclidean distance between pixel p(x,y) and neighboring pixel q(i,j), ||I(p)-I(q)|| is the brightness difference between pixels p and q in the original light field image, D(q) is the original disparity value or depth value of the neighboring pixel q(i,j), and Ω is the neighborhood window of the current pixel p(x,y).

7. The light field image super-resolution processing method according to claim 1, characterized in that: The process of performing super-resolution processing on the depth image and performing edge sharpening and texture generation is as follows: Performing non-uniform upsampling on the depth map, and extracting features using a multi-scale feature fusion network to achieve depth map fusion and feature enhancement; A gradient difference term is introduced into the loss function, and based on the gradient difference term, the edge alignment between the super-resolved image and the high-resolution real image is forced to be completed to achieve edge sharpening; The sharpened depth map is spliced with the low-resolution image and input into the generator to output a high-resolution image.

8. The light field image super-resolution processing method according to claim 7, characterized in that: The process of performing non-uniform upsampling on the depth map and extracting features using a multi-scale feature fusion network to achieve depth map fusion and feature enhancement is as follows: Divide the depth map into foreground, midground, and background, upsample the foreground area by a factor of 2, upsample the midground area by a factor of 1.5, and upsample the background area by a factor of 1 or 1.2; The low-level and high-level features of the low-resolution image are extracted through the residual network, and the depth map is input as an additional channel. It is spliced with the low-level and high-level features or weightedly fused through the attention mechanism to complete the depth map fusion and feature enhancement.

9. A light field image super-resolution processing system, used to implement the light field image super-resolution processing method according to any one of claims 1 to 8, characterized in that: include: Light field data acquisition module (1), light field data analysis module (2), multi-view restoration module (3), depth estimation module (4) and super-resolution generation module (5); The light field data acquisition module (1) is used to acquire a degraded light field image, acquire an original light field image through a photographic acquisition device, and obtain a multi-view sub-aperture image; The light field data parsing module (2) is used to extract and align the multi-view sub-aperture images, extract the SAI sequences of each view through a light field decoding tool, and align all SAI sequences to a central view coordinate system through an optical flow method or a geometric transformation method; The multi-view restoration module (3) is used to extract cross-view features of the SAI sequence, complete the restoration by calculating the attention weight and performing denoising processing, and optimize the restored SAI sequence to obtain an optimized SAI sequence; The depth estimation module (4) is used to perform depth estimation on the optimized SAI sequence, generate a disparity map and perform depth map conversion to obtain a depth image; The super-resolution generation module (5) is used to perform super-resolution processing on the depth image, and perform edge sharpening and texture generation.

Images