Light field image super-resolution reconstruction method based on sub-pixel and gradient guidance
Through the super-resolution reconstruction method of light field images based on subpixel and gradient guidance, multiple subpixel features are extracted and fused, and the problems of pixel differences and edge details loss in light field images are solved, thereby achieving higher quality light field images Super resolution reconstruction.
Patent Information
- Application Number
- CN202310874018.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-17
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-07-17
AI Technical Summary
Existing super-resolution reconstruction methods for light field images are difficult to effectively solve the problem of pixel differences in light field images, resulting in artifacts and distortions in the reconstruction image, and failure to effectively protect edge details.
Using a super-resolution reconstruction method of light field images based on subpixel and gradient guidance, multiple subpixel features of light field and gradient are extracted through multiple subpixel information extraction and gradient prior guidance, and feature representation enhancement and fusion are performed for super-superior use. Resolution reconstruction.
It improves the effect and quality of super-resolution reconstruction of light field images, makes up for the problem of pixel differences, protects edge texture details, and significantly improves the authenticity and clarity of the reconstructed image.
Smart Images

Figure CN117114987B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of light field image super-resolution reconstruction, and in particular to a light field image super-resolution reconstruction method based on sub-pixel and gradient guidance. Background Art
[0002] Light field contains both spatial and angular information of the scene, and has broad application prospects in fields such as computer vision and computer graphics. It can be used for target detection, depth estimation, and three-dimensional (3D) reconstruction. As a high-dimensional data, light field is difficult to represent in the 3D world and requires a special light field camera to record it. Compared with ordinary cameras, light field cameras embed a microlens array between the main lens and the image sensor, which records the direction and intensity of spatial light from multiple angles at the same time, providing users with richer scene and motion information. Due to the limitations of the microlens array in the light field camera, the resolution of the obtained light field image is generally low, which hinders the application of light field. Therefore, a large number of studies on light field image super-resolution (SR) have emerged.
[0003] Existing light field image SR methods are mainly divided into traditional SR methods and deep learning-based SR methods. Traditional light field image SR methods use prior disparity information to explicitly project viewpoint image pixel information to achieve the purpose of SR based on the correspondence between light field viewpoint image disparity and pixel points. However, the disparity prior information is heavily dependent on the quality of the viewpoint image itself, and existing disparity estimation methods usually cannot meet the high precision requirements of such methods for disparity prior information. Light field image SR methods based on deep learning do not require prior disparity information. By constructing a mapping relationship from low-resolution images to high-resolution images, they can directly learn light field information through the network to achieve good SR quality.
[0004] The shape and position of objects in light field images will change with different viewing angles. Therefore, there are pixel differences between views in light field images. This difference is mainly reflected in the difference in the direction and intensity of the light of the pixels. Ignoring these differences in the SR process will cause artifacts and distortion in the reconstructed image. However, existing deep learning-based methods mainly obtain a large receptive field to cover the difference range by cascading multiple convolutional layers, which cannot solve the problem of pixel differences between views during light field image SR. At the same time, there is a problem of detail loss in the light field image SR process. Most of the existing methods do not pay attention to protecting edge details during the reconstruction process, resulting in poor effect and quality of light field image super-resolution reconstruction. Therefore, how to improve the effect and quality of light field image super-resolution reconstruction is a technical problem that needs to be solved urgently. Summary of the invention
[0005] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide a light field image super-resolution reconstruction method based on sub-pixel and gradient guidance, which can compensate for the pixel differences in the light field through multiple sub-pixel information, and can guide the super-resolution reconstruction through gradient prior to protect the edge texture details of the reconstructed image, thereby improving the effect and quality of light field image super-resolution reconstruction.
[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0007] The light field image super-resolution reconstruction method based on sub-pixel and gradient guidance includes:
[0008] S1: Acquire the target light field image to be reconstructed;
[0009] S2: Input the target light field image into the trained super-resolution reconstruction model and output the corresponding super-resolution reconstructed image;
[0010] The super-resolution reconstruction model is trained through the following steps:
[0011] S201: first, converting a high-resolution light field image used as a sample into a low-resolution light field image; then extracting a gradient map of the low-resolution light field image to obtain a corresponding gradient array; finally, using the low-resolution light field image and the corresponding gradient array as model input;
[0012] S202: extracting the low-resolution light field image and the multiple sub-pixel features of the corresponding gradient array through a multiple sub-pixel information extraction module to obtain corresponding light field multiple sub-pixel features and gradient multiple sub-pixel features;
[0013] S203: performing feature representation enhancement on the light field multiple sub-pixel feature and the gradient multiple sub-pixel feature through an enhancement module to obtain corresponding light field multiple sub-pixel enhancement features and gradient multiple sub-pixel enhancement features;
[0014] S204: performing feature fusion on the light field multiple sub-pixel enhancement feature and the gradient multiple sub-pixel enhancement feature through a fusion module to obtain a corresponding multiple sub-pixel fusion feature;
[0015] S205: performing super-resolution reconstruction based on multiple sub-pixel fusion features through a reconstruction module to obtain a reconstructed image; then adding the low-resolution light field image to the reconstructed image after bicubic interpolation to generate a corresponding super-resolution reconstructed image;
[0016] S206: calculating a training loss according to the super-resolution reconstructed image and the high-resolution light field image used as a sample, and optimizing parameters of the super-resolution reconstruction model based on the training loss;
[0017] S207: repeating steps S201 to S206 until the super-resolution reconstruction model converges;
[0018] S3: Using the output super-resolution reconstructed image as the super-resolution reconstruction result of the target light field image.
[0019] Preferably, the low-resolution light field image or gradient array is stacked into four image piles in the horizontal, vertical, diagonal and anti-diagonal directions respectively and input into the multiple sub-pixel information extraction module: first, the sub-pixel feature information of the four image piles is extracted respectively; then the sub-pixel feature information of each image pile is connected to obtain the global sub-pixel feature; finally, the global sub-pixel feature is reduced in dimension and aligned in feature to obtain the corresponding multiple sub-pixel features.
[0020] Preferably, the multiple sub-pixel information extraction module includes four feature multiple sub-pixel information extraction modules that respectively take four image stacks as input and are used to extract sub-pixel feature information, a connection module for connecting the sub-pixel feature information output by the four feature multiple sub-pixel information extraction modules, a 3D convolution module for reducing the dimension of the connected global sub-pixel features, and an angle alignment module for feature alignment of the reduced global sub-pixel features; the output of the angle alignment module is the corresponding multiple sub-pixel features.
[0021] Preferably, the feature multiple sub-pixel information extraction module includes a 3D convolution layer and four stacked residual blocks connected end to end in sequence, and the input of the first residual block and the output of the last residual block are summed up as the sub-pixel feature information output by the feature multiple sub-pixel information extraction module; each residual block includes a PReLu activation layer and a 3D convolution layer connected end to end in sequence, and the input of the PReLu activation layer is summed up with the output of the 3D convolution layer.
[0022] Preferably, the light field multiple sub-pixel features or the gradient multiple sub-pixel features are converted into corresponding feature sequences and then input into the enhancement module: first, the sub-pixel spatial features of the feature sequence are extracted; then, the corresponding query matrix, keyword matrix and value matrix are generated based on the sub-pixel spatial features, and a spatial feature map is generated according to the three matrices; finally, the spatial feature map is added to the input feature sequence to obtain the corresponding multiple sub-pixel enhanced features.
[0023] Preferably, the enhancement module includes the following processing steps:
[0024] S2031: extracting sub-pixel spatial features of the feature sequence respectively through three convolutional layer branches;
[0025] S2032: performing expansion operations on the three sub-pixel spatial features respectively to obtain corresponding local feature blocks; then reshaping the three local feature blocks into corresponding query matrix Q1, keyword matrix K1 and value matrix V1 respectively;
[0026] S2033: Perform a transposition operation on K1 to obtain the keyword matrix transposition K1 T ;
[0027] S2034: K1 T After multiplying with Q1, it is multiplied by V1, and then the resulting product is folded and convolved to obtain a spatial feature map;
[0028] S2035: Add the spatial feature map to the input feature sequence to obtain corresponding multiple sub-pixel enhanced features.
[0029] Preferably, the light field multiple sub-pixel enhancement features and the gradient multiple sub-pixel enhancement features are input into a fusion module: first, the light field enhancement spatial features and the gradient enhancement spatial features of the light field multiple sub-pixel enhancement features and the gradient multiple sub-pixel enhancement features are extracted; then, based on the light field enhancement spatial features and the gradient enhancement spatial features, corresponding query matrices, keyword matrices and value matrices are generated, and an enhancement feature map is generated according to the three matrices; finally, the enhancement feature map is added to the input light field multiple sub-pixel enhancement features to obtain corresponding multiple sub-pixel fusion features.
[0030] Preferably, the fusion module includes the following processing steps:
[0031] S2041: Extracting light field enhancement spatial features of multiple sub-pixel enhancement features of the light field through a convolutional layer branch;
[0032] S2042: respectively extracting gradient enhanced spatial features of gradient multiple sub-pixel enhanced features through two convolutional layer branches;
[0033] S2043: performing an expansion operation on the light field enhancement spatial feature to obtain a corresponding local feature block; and then reshaping the local feature block into a corresponding query matrix Q2;
[0034] S2044: performing expansion operations on the two gradient enhanced spatial features respectively to obtain corresponding local feature blocks; then reshaping the two local feature blocks into corresponding keyword matrices K2 and value matrices V2 respectively;
[0035] S2045: Perform a transposition operation on K2 to obtain the keyword matrix transposition K2 T ;
[0036] S2046: K2 T After multiplying with Q2, it is multiplied with V2 again, and then the product is folded and convolved to obtain an enhanced feature map;
[0037] S2047: Add the enhanced feature map to the input light field multiple sub-pixel enhanced feature to obtain a corresponding multiple sub-pixel fusion feature.
[0038] Preferably, the reconstruction module is composed of an information multi-distillation module and an upsampling module, wherein the upsampling module comprises two convolutional layers and a pixel reorganization layer;
[0039] When the reconstruction module performs super-resolution reconstruction: first, the multi-level feature information is extracted from the multiple sub-pixel fusion features through the information multi-distillation module; then the channel dimension of the multi-level feature information is increased to a through the first convolutional layer. 2 N, and the corresponding reconstructed features are obtained; then the reconstructed features are upsampled to the target resolution aH×aW through the pixel recombination layer; finally, the channel dimension of the upsampled reconstructed features is reduced to 1 through the second convolutional layer to obtain the corresponding reconstructed image.
[0040] Preferably, the training loss is calculated by the following formula:
[0041]
[0042] Where: loss(x,y) represents the training loss; f(x i ) represents the generated super-resolution reconstructed image, x i represents the input low-resolution light field image; y i represents the high-resolution light field image used as a sample; n represents the number of samples.
[0043] Compared with the prior art, the light field image super-resolution reconstruction method based on sub-pixel and gradient guidance in the present invention has the following beneficial effects:
[0044] The present invention performs super-resolution reconstruction on a light field image through a trained super-resolution reconstruction model. The model first uses a low-resolution light field image and a corresponding gradient array as model input, then extracts sub-pixel feature information to obtain multiple sub-pixel features of the light field and gradient, then performs feature representation enhancement on the multiple sub-pixel features of the light field and gradient, and finally fuses the multiple sub-pixel features of the light field and gradient after the feature representation enhancement, and performs super-resolution reconstruction based on the multiple sub-pixel fusion features, and then adds the reconstructed image to the low-resolution light field image to obtain a super-resolution reconstructed image. On the one hand, the present invention extracts the multiple sub-pixel features of the light field of the light field image (i.e., all sub-aperture images), and performs feature representation enhancement on the multiple sub-pixel features of the light field, and compensates for the pixel differences in the light field through the multiple sub-pixel features of the light field after the feature representation enhancement, so that a higher quality reconstructed image can be reconstructed while improving the spatial resolution of the light field image, and improves the problems of distortion, artifacts, etc. of the reconstructed image caused by ignoring pixel differences in the super-resolution reconstruction process, thereby improving the effect and quality of super-resolution reconstruction of the light field image. On the other hand, the present invention extracts the gradient map of the light field image to obtain a gradient array on the basis of extracting the light field multiple sub-pixel features, and then extracts the sub-pixel feature information of the gradient array (that is, all gradient maps) to obtain the gradient multiple sub-pixel features, and fuses the gradient multiple sub-pixel features after the feature representation enhancement with the light field multiple sub-pixel features, so that the gradient multiple sub-pixel features can be used as gradient priors to guide the super-resolution reconstruction of the light field multiple sub-pixel features, thereby protecting the edge texture details of the super-resolution reconstructed image, thereby further improving the effect and quality of super-resolution reconstruction of the light field image. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:
[0046] Figure 1 The logical block diagram of the light field image super-resolution reconstruction method based on sub-pixel and gradient guidance;
[0047] Figure 2 This is a network structure diagram of the light field image super-resolution reconstruction model;
[0048] Figure 3 It is a schematic diagram of sub-pixel information in a horizontal polar plane image;
[0049] Figure 4 This is the network structure diagram of the multiple sub-pixel information extraction module;
[0050] Figure 5 This is the network structure diagram of the enhancement module;
[0051] Figure 6 This is the network structure diagram of the fusion module;
[0052] Figure 7 Comparison of visual effects of different methods in the scene Building_Decoded;
[0053] Figure 8 Comparison of visual effects of different methods under scene Origami. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.
[0055] It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. In the description of the present invention, it should be noted that the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the invention product is usually placed when used, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. In addition, the terms "horizontal", "vertical", etc. do not mean that the components are required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0056] The following is a further detailed description through specific implementation methods:
[0057] Example:
[0058] This embodiment discloses a light field image super-resolution reconstruction method based on sub-pixel and gradient guidance.
[0059] like Figure 1 As shown, the light field image super-resolution reconstruction method based on sub-pixel and gradient guidance includes:
[0060] S1: Acquire the target light field image to be reconstructed;
[0061] The light field image super-resolution (Super-Resolution, SR) described in the present invention refers to a technology that uses image processing and machine learning techniques to reconstruct a high-resolution (High-Resolution, HR) image from an existing low-resolution (Low-Resolution, LR) image. Specifically, a light field image can be represented as a macro-pixel image, a sub-aperture image (Sub-aperture Image, SAI) and an epipolar plane image (Epi), and the three images can be converted into each other. The present invention represents the light field image as a sub-aperture image array. The light field can be represented as L(x, y, s, t), where (x, y) represents the spatial dimension, (s, t) represents the angular dimension, and L∈R (H×W×S×T) The light field image can be viewed as an array of S×T sub-aperture images, each with a resolution of H×W pixels. The goal of light field image super-resolution reconstruction is to reconstruct the image from a low-resolution (LR) light field L LR (x, y, s, t) reconstructs the super-resolution (SR) light field L SR (x,y,s,t),L SR ∈R (aH×aW×S×T) , where a is the spatial magnification. The formula is described as:
[0062] L SR =f(L LR ), where f represents the function mapping from the LR light field to the HR light field.
[0063] S2: Input the target light field image into the trained super-resolution reconstruction model and output the corresponding super-resolution reconstructed image;
[0064] Combination Figure 2 As shown, the super-resolution reconstruction model is trained through the following steps:
[0065] S201: first, convert the high-resolution light field image used as a sample into a low-resolution light field image; then extract the gradient map of the low-resolution light field image (i.e., all sub-aperture images) to obtain a corresponding gradient array; finally, use the low-resolution light field image and the corresponding gradient array as model input;
[0066] When training a super-resolution reconstruction model, the present invention performs bicubic downsampling on a high-resolution light field image to obtain a low-resolution image, so that the low-resolution light field image in real life can be better simulated, thereby improving the practical application effect of the trained super-resolution reconstruction model, thereby improving the practicability of super-resolution reconstruction of light field images.
[0067] S202: extracting multiple sub-pixel features of the low-resolution light field image and the corresponding gradient array (i.e., all gradient maps) through a multiple sub-pixel information extraction module (Multiple Sub-Pixel Information Extraction, MSPIE) to obtain corresponding light field multiple sub-pixel features and gradient multiple sub-pixel features;
[0068] S203: performing feature representation enhancement on the light field multiple sub-pixel feature and the gradient multiple sub-pixel feature through a (feature representation) enhancement module to obtain corresponding light field multiple sub-pixel enhancement features and gradient multiple sub-pixel enhancement features;
[0069] S204: performing feature fusion on the light field multiple sub-pixel enhancement feature and the gradient multiple sub-pixel enhancement feature through a fusion module to obtain a corresponding multiple sub-pixel fusion feature;
[0070] S205: performing super-resolution reconstruction based on multiple sub-pixel fusion features through a reconstruction module to obtain a reconstructed image; then adding the low-resolution light field image to the reconstructed image after bicubic interpolation (completing global residual learning) to generate a corresponding super-resolution reconstructed image;
[0071] In this embodiment, the bicubic interpolation used is a commonly used image interpolation method, which is used to increase the size of the low-resolution light field image to be consistent with the reconstructed image. It performs interpolation on a two-dimensional plane and uses a cubic polynomial to approximate the pixel values in the image.
[0072] S206: calculating a training loss according to the super-resolution reconstructed image and the high-resolution light field image used as a sample, and optimizing parameters of the super-resolution reconstruction model based on the training loss;
[0073] S207: repeating steps S201 to S206 until the super-resolution reconstruction model converges;
[0074] S3: Using the output super-resolution reconstructed image as the super-resolution reconstruction result of the target light field image.
[0075] The present invention performs super-resolution reconstruction on a light field image through a trained super-resolution reconstruction model. The model first uses a low-resolution light field image and a corresponding gradient array as model input, then extracts sub-pixel feature information to obtain multiple sub-pixel features of the light field and gradient, then performs feature representation enhancement on the multiple sub-pixel features of the light field and gradient, and finally fuses the multiple sub-pixel features of the light field and gradient after the feature representation enhancement, and performs super-resolution reconstruction based on the multiple sub-pixel fusion features, and then adds the reconstructed image to the low-resolution light field image to obtain a super-resolution reconstructed image. On the one hand, the present invention extracts the multiple sub-pixel features of the light field of the light field image (i.e., all sub-aperture images), and performs feature representation enhancement on the multiple sub-pixel features of the light field, and compensates for the pixel differences in the light field through the multiple sub-pixel features of the light field after the feature representation enhancement, so that a higher quality reconstructed image can be reconstructed while improving the spatial resolution of the light field image, and improves the problems of distortion, artifacts, etc. of the reconstructed image caused by ignoring pixel differences in the super-resolution reconstruction process, thereby improving the effect and quality of super-resolution reconstruction of the light field image. On the other hand, the present invention extracts the gradient map of the light field image to obtain a gradient array on the basis of extracting the light field multiple sub-pixel features, and then extracts the sub-pixel feature information of the gradient array (that is, all gradient maps) to obtain the gradient multiple sub-pixel features, and fuses the gradient multiple sub-pixel features after the feature representation enhancement with the light field multiple sub-pixel features, so that the gradient multiple sub-pixel features can be used as gradient priors to guide the super-resolution reconstruction of the light field multiple sub-pixel features, thereby protecting the edge texture details of the super-resolution reconstructed image, thereby further improving the effect and quality of super-resolution reconstruction of the light field image.
[0076] 1. Multiple sub-pixel information extraction
[0077] Unlike traditional multi-view images, the views in light field images have different angular directions. The view in one angular direction contains the sub-pixel offset of the central view in the corresponding spatial direction. The sub-pixel offsets in different spatial directions can be found in the views in the corresponding angular directions. According to the image consistency of the light field, if the pixel difference between adjacent views is 1 / 3, the area can be upsampled and enlarged by 3 times by extracting the corresponding pixels from the surrounding 3 views. Therefore, the sub-pixel offsets between different views are found based on the difference information, thereby improving the spatial resolution of each view.
[0078] By fixing an angular dimension (s, t) and spatial dimension (x, y) of the light field image L, the corresponding EPI (epiplane image) can be obtained. EPI contains information in both spatial and angular dimensions, and can effectively reflect the geometric consistency of the light field scene. In addition, the slope of the lines in the EPI reflects the difference in the corresponding pixels, so the difference problem in the light field SR can be solved by using EPI. The SAI is stacked into an image stack along a fixed direction, and the two-dimensional slice of the image stack along a certain direction is the EPI in the corresponding direction. To further explain the connection between the spatial dimension and the angular dimension, Figure 3 The sub-pixel information in the horizontal EPI is shown in Figure 1. The two-dimensional slice E can be obtained by stacking SAI along the horizontal direction and then slicing it along the horizontal direction. H (x,u) is the horizontal EPI. The pixels in different views of the light field image can be represented as:
[0079] L(x,y,s,t)=L(x+Δx,y+Δy,s+Δs,t+Δt);
[0080] Where: Δx = Δs × d, Δy = Δt × d, d is the pixel difference between the reference point and other points.
[0081] Based on d, a pixel in one view can be matched with another pixel in another view. Due to the dense sampling in the angle dimension, sub-pixel information in one view can be found in other views. Figure 3 As shown in Figure 1, the horizontal sub-pixel information of the reference point p can be found in the two surrounding views. p (x p ,y p ,s p +Δx / d,t p ) and L p' (x p +Δx,y p ,s p ,t p ), indicating that the corresponding horizontal sub-pixel information can be extracted from the adjacent view. Figure 3 In the horizontal EPI in , the reference point p corresponds to p′ and p″. In addition, the vertical and diagonal sub-pixel information can be found in the corresponding vertical and diagonal EPIs, respectively. Once the surrounding sub-pixel information in the four directions in the central EPI is found, the corresponding area can complete SR.
[0082] Specifically, the present invention converts the low-resolution light field image (i.e., all sub-aperture images) L LR ∈R (H×W×S×T) Or the gradient array (i.e. all gradient maps) G LR ∈R(H×W×S×T) Four image piles are stacked in horizontal, vertical, diagonal and anti-diagonal directions and input into the multiple sub-pixel information extraction module: first, the sub-pixel feature information of the four image piles is extracted respectively; then the sub-pixel feature information of each image pile is connected to obtain the global sub-pixel feature; finally, the global sub-pixel feature is reduced in dimension and aligned to obtain the corresponding multiple sub-pixel feature (i.e., the light field multiple sub-pixel feature F MSP ∈R (H×W×S×T×C) Or gradient multiple sub-pixel features G MSP ∈R (H×W×S×T×C) ). The image stacks in horizontal, vertical, diagonal and anti-diagonal directions are represented as:
[0083]
[0084]
[0085]
[0086]
[0087] Where: m∈{1,...,A}. When 1≤i≤A, 1≤m≤A; when i>A, iA≤m≤2A-i. There are A images in the horizontal and vertical image piles, each image pile has A images, and there are 2A-1 image piles in the diagonal direction, and the number of images in each image pile is different.
[0088] Combination Figure 4 As shown, the multiple sub-pixel information extraction module includes four (parallel arranged) feature multiple sub-pixel information extraction modules that respectively take four image stacks as input and are used to extract sub-pixel feature information, a connection module for connecting the sub-pixel feature information output by the four feature multiple sub-pixel information extraction modules to generate a global sub-pixel feature, a 3D convolution module for reducing the dimension of the connected global sub-pixel feature, and an Angular Alignment Module (AAM) for feature alignment of the reduced global sub-pixel feature; the output of the Angular Alignment Module is the corresponding multiple sub-pixel feature. The structure of the multiple sub-pixel information extraction module can make up for the pixel difference problem in light field image super-resolution.
[0089] Specifically, the feature multiple sub-pixel information extraction module includes a 3D convolution layer and four stacked residual blocks connected end to end in sequence, and the input of the first residual block and the output of the last residual block are summed as the sub-pixel feature information output by the feature multiple sub-pixel information extraction module. Each residual block includes a PReLu activation layer and a 3D convolution layer connected end to end in sequence, and the input of the PReLu activation layer is summed with the output of the 3D convolution layer. Among them, the 3D convolution can extract the sub-pixel information in the corresponding direction EPI, and the structure of the feature multiple sub-pixel information extraction module can ensure a sufficiently large receptive field to extract sub-pixel information.
[0090] The output features of the feature multiple sub-pixel information extraction module are:
[0091]
[0092] Among them, when θ∈{0°,90°}, i∈{1,2,...,A}, when θ∈{45°,135°}, i∈{1,2,...,2A-1}.
[0093] Will Reshape into 5-dimensional data F θ ∈R (H×W×S×T×C) , C is the feature channel. The features in the four directions are connected, and the dimension is reduced through a convolution layer. Finally, the features are aligned through AAM to obtain multiple sub-pixel features F MSP ∈R (H×W×S×T×C) .
[0094] In this embodiment, the angle-variable alignment module disclosed in "Wang YQ, Yang JG, Wang LG, et al. Light field image super-resolution using deformable convolution" is improved. Since only the convolution features need to be aligned, the variable convolution is removed. Since EPI can be regarded as a two-dimensional slice along the direction of the image stack, the 3D convolution in the feature extraction block can extract the sub-pixel information in the EPI in the corresponding direction. In order to ensure a large enough receptive field to extract sub-pixel information, the convolution kernel size is set to 3×3×3, and each feature extraction block contains four residual blocks.
[0095] In order to extract sub-pixel feature information of low-resolution light field images, the present invention constructs an MSPIE module, which divides the sub-aperture image into four image piles: horizontal, vertical, diagonal and anti-diagonal, and extracts the sub-pixel feature information of each image pile respectively, so that multiple sub-pixel features of the light field image and the gradient array can be effectively extracted, which provides effective data support for subsequent feature fusion and super-resolution reconstruction, thereby further improving the effect and quality of super-resolution reconstruction of light field images.
[0096] 2. Feature Representation Enhancement
[0097] Considering that convolutional features can only capture local context information within each SAI independently, there is a lack of global context information between different SAIs. Unlike traditional convolution, the visual Transformer regards an image as a sequence of symbols and establishes the relationship between all symbols through the self-attention mechanism. Therefore, adding the Transformer can further enrich the feature representation. Since the light field image is composed of multiple views and each view is highly correlated, the light field image can be regarded as a continuous sequence. The present invention introduces the Transformer in the video SR into the light field image SR for feature representation enhancement to better capture the global context information between views. The Transformer includes 4 spatial angle enhanced self-attention layers.
[0098] Specifically, the present invention converts the light field multiple sub-pixel features or the gradient multiple sub-pixel features into the corresponding feature sequence F s ∈R (L×H×W×N) Post-input enhancement module: first, extract the sub-pixel spatial features of the feature sequence; then generate the corresponding query matrix, keyword matrix and value matrix based on the sub-pixel spatial features, and generate a spatial feature map based on the three matrices; finally, add the spatial feature map to the input feature sequence to obtain the corresponding multiple sub-pixel enhanced features.
[0099] Combination Figure 5 As shown, the enhancement module includes the following processing steps:
[0100] Consider all SAIs as a sequence, the characteristic sequence F s ∈R (L×H×W×N) As the input of Transformer, where L is the sequence length and N=4C.
[0101] S2031: extract feature sequences (such as feature sequences F of multiple sub-pixel features of light fields) through three convolutional layer branches respectively. s ∈R (L×H×W×N) )’s sub-pixel spatial features;
[0102] S2032: performing expansion operations on the three sub-pixel spatial features respectively to obtain corresponding local feature blocks; then reshaping the three local feature blocks into corresponding query matrix Q1, keyword matrix K1 and value matrix V1 respectively;
[0103] The expansion operation in this embodiment is a method of converting a feature map from a three-dimensional tensor (usually spatial size × number of channels) to a two-dimensional matrix. Local feature blocks refer to dividing the feature map into different blocks or sub-regions, and treating these blocks as local features. Steps of expansion operation and reshaping feature matrix: 1) Define local feature blocks: Determine how to divide the feature map into local feature blocks based on task requirements and input feature map size, which can be a regular grid segmentation or any irregular method. 2) Divide feature map: Divide the feature map into different blocks according to the defined local feature blocks, which can be achieved by using sliding windows or other division techniques. 3) Expand local feature blocks: For each local feature block, expand it into a one-dimensional vector, and a two-dimensional matrix can be obtained by expanding all pixels of each block into a row. 4) Concatenate feature blocks: Concatenate all expanded local feature blocks in a certain order to form a feature matrix. Usually, each row of the feature matrix corresponds to an expanded local feature block. 5) Output feature matrix: Get the reshaped feature matrix as the result.
[0104] S2033: Perform a transposition operation on K1 to obtain the keyword matrix transposition K1 T ;
[0105] S2034: K1 T After multiplying with Q1 and then with V1, the resulting product is folded and convolved (1×1 convolutional layer) to obtain a spatial feature map;
[0106] S2035: Add the spatial feature map to the input feature sequence to obtain the corresponding multiple sub-pixel enhancement feature (such as the light field multiple sub-pixel enhancement feature F T ∈R (L×H×W×N) Or gradient multiple sub-pixel enhancement feature G T ∈R (L×H×W×N) ).
[0107] In order to enrich the global context information between different SAIs, the present invention introduces Transformer to enhance the feature representation of multiple sub-pixel features. At the same time, the enhancement module of the above structure can better capture the global context information between views, thereby strengthening the connection between the spatial dimension and the angular dimension.
[0108] 3. Gradient feature fusion
[0109] In order to better protect the edge details of the reconstructed image during the SR process, the present invention uses gradients to guide the light field image SR. First, the gradient maps of all SAIs are extracted to obtain the gradient array G LR ∈R (H×W×S×T) Then use the MSIPE module to extract the gradient multiple sub-pixel features G MSP ∈R (H×W×S×T×C) , and then enhance the feature representation to obtain the enhanced gradient feature G T ∈R (L×H×W×N) The light field image enhancement features and gradient image enhancement features are fused to enhance the structural details. The spatial angle enhancement self-attention layer focuses on the relationship with the same sequence, while the fusion layer needs to focus on the relationship between the gradient sequence and the original feature sequence. Therefore, the spatial angle enhancement self-attention layer is improved to obtain the gradient fusion self-attention layer.
[0110] Specifically, the light field multiple sub-pixel enhancement features and the gradient multiple sub-pixel enhancement features are input into the fusion module (gradient fusion attention layer): the light field multiple sub-pixel enhancement features and the gradient multiple sub-pixel enhancement features are input into the fusion module: first, the light field enhancement spatial features and the gradient enhancement spatial features of the light field multiple sub-pixel enhancement features and the gradient multiple sub-pixel enhancement features are extracted; then, the corresponding query matrix, keyword matrix and value matrix are generated based on the light field enhancement spatial features and the gradient enhancement spatial features, and an enhanced feature map is generated according to the three matrices; finally, the enhanced feature map is added to the input light field multiple sub-pixel enhancement features to obtain the corresponding multiple sub-pixel fusion features.
[0111] Combination Figure 6 As shown, the fusion module includes the following processing steps:
[0112] S2041: Extracting light field enhancement spatial features of multiple sub-pixel enhancement features of the light field through a convolutional layer branch;
[0113] S2042: respectively extracting gradient enhanced spatial features of gradient multiple sub-pixel enhanced features through two convolutional layer branches;
[0114] S2043: performing an expansion operation on the light field enhancement spatial feature to obtain a corresponding local feature block; and then reshaping the local feature block into a corresponding query matrix Q2;
[0115] S2044: performing expansion operations on the two gradient enhanced spatial features respectively to obtain corresponding local feature blocks; then reshaping the two local feature blocks into corresponding keyword matrices K2 and value matrices V2 respectively;
[0116] S2045: Perform a transposition operation on K2 to obtain the keyword matrix transposition K2 T ;
[0117] S2046: K2 T After multiplying with Q2, it is multiplied with V2 again, and then the resulting product is folded and convolved (1×1 convolution layer) to obtain an enhanced feature map;
[0118] S2047: Add the enhanced feature map to the input light field multiple sub-pixel enhanced feature to obtain the corresponding multiple sub-pixel fusion feature F f ∈R (L×H×W×N) .
[0119] In order to solve the problem of edge detail loss in the reconstruction process of light field images, the present invention uses gradient-guided super-resolution reconstruction process. Through ablation experiments, it is shown that gradient-guided light field SR can effectively protect the edge details of the image. At the same time, the fusion module of the structure better protects the edge details of the reconstructed image by fusing gradients.
[0120] 4. Super-resolution reconstruction
[0121] The reconstruction module consists of an information multiple distillation module and an upsampling module, where the upsampling module contains two (1×1) convolutional layers and a pixel reorganization layer. When the reconstruction module performs super-resolution reconstruction: first, the information multiple distillation module is used to extract multi-level feature information from multiple sub-pixel fusion features; then, the first 1×1 convolutional layer is used to increase the channel dimension of the multi-level feature information to a 2 N, and the corresponding reconstructed features are obtained; then the reconstructed features are upsampled to the target resolution aH×aW through the pixel recombination layer; finally, the channel dimension of the upsampled reconstructed features is reduced to 1 through the second 1×1 convolutional layer to obtain the corresponding reconstructed image.
[0122] In this embodiment, the information multi-distillation module is generally expressed as a residual structure, extracting a portion of useful features each time, and the remaining features continue to be extracted through convolution, and finally the features extracted at each step are connected, and then passed through a channel attention layer, and finally the dimensionality is reduced by 1*1 convolution. Specifically, the information multi-distillation module can adopt the information multi-distillation module disclosed in "Hui Z, Gao XB, Yang YC, et al. Lightweight image super-resolution withinformation multi-distillation network".
[0123] The reconstructed image generated by the reconstruction module of the present invention can maintain the consistency of details and structures at multiple levels, and reduces the computational complexity and the number of parameters.
[0124] In the specific implementation process, the five public databases used for training and testing are EPFL, HCI_new, HCI_old, INRIA_Lytro and Stanford_Gantry. 144 scenes are selected for training and 23 scenes are used for testing. The original angular resolution of all light field images is 9×9. During training and testing, the original light field images are cropped to a suitable resolution with an angular resolution of 5×5. The original light field images are cropped into 64×64 image blocks with a stride of 32, and then the image blocks are downsampled to 32×32 using bicubic interpolation as the low-resolution input image of the network.
[0125] The super-resolution reconstruction model was trained on a PC with an NVIDIA RTX 4090 GPU based on the Pytorch framework. First, the light field image was converted from RGB three-channel image to YCbCr color space, and the Y channel image was extracted for super-resolution. When generating visual contrast results, the Cb and Cr channels were upsampled using bicubic interpolation. During training, the original light field image was cropped into 64×64 image blocks with a stride of 32, and then the image blocks were downsampled to 32×32 using bicubic interpolation as the LR input image of the network. The cropped training data was randomly flipped horizontally, flipped vertically, and rotated 90 degrees to increase the training data. The number of convolution kernels in the 3D convolution layer was 32, and the convolution kernel size was 3×3×3. The initial learning rate was set to 2e-4, and the learning rate optimization method was ADAM, β1=0.9, β2=0.999. The learning rate was halved every 15 epochs, and a total of 59 epochs were trained with a batch size of 8 and a mean absolute error L1 loss function.
[0126] The training loss is calculated using the following formula:
[0127]
[0128] Where: loss(x,y) represents the training loss; f(x i ) represents the generated super-resolution reconstructed image, x i represents the input low-resolution light field image; y i represents the high-resolution light field image used as a sample; n represents the number of samples.
[0129] The present invention calculates the training loss through the above formula, which has a stable gradient for any input value, does not cause the gradient explosion problem, and has a more robust solution.
[0130] In order to better illustrate the advantages of the technical solution of the present invention, the following experiments are disclosed in this embodiment.
[0131] In this experiment, the five public databases used for training and testing are EPFL (from Rerabek M, Ebrahimi T. New light field image dataset), HCI_new (Honauer K, Johannsen O, Kondermann D, et al. A dataset and evaluation methodology for depth estimation on 4D light fields), HCI_old (from Wanner S, Meister S, Goldluecke B. Datasets and benchmarks for densely sampled 4D light fields), INRIA_Lytro (from Pendu ML, Jiang XR, Guillemot C. Light field in painting propagation via low rank matrix completion) and Stanford_Gantry (from Vaish V, Adams A. The (new) standford lightfield archive). 144 scenes were selected for training and 23 scenes for testing. The original angular resolution of all light field images is 9 × 9. During training and testing, the original light field images are cropped to a suitable resolution of 5 × 5 and 64 × 64. The original images are downsampled using bicubic interpolation to generate LR images.
[0132] 1. Training details
[0133] In this experiment, the designed super-resolution reconstruction model was trained based on the Pytorch framework on a PC with an NVIDIA RTX 4090 GPU. First, the light field image was converted from an RGB three-channel image to a YCbCr color space, and the Y channel image was extracted for super-resolution. When generating visual contrast results, the Cb and Cr channels were upsampled using bicubic interpolation. During training, the original light field image was cropped into 64×64 image blocks with a stride of 32, and then the image blocks were downsampled to 32×32 using bicubic interpolation as the LR input image of the network. The cropped training data was randomly flipped horizontally, flipped vertically, and rotated 90 degrees to increase the training data. The number of convolution kernels in the 3D convolution layer was 32, and the convolution kernel size was 3×3×3. The initial learning rate was set to 2e-4, and the learning rate optimization method was ADAM, β1=0.9, β2=0.999. The learning rate is halved every 15 epochs, and a total of 59 epochs are trained. The batch size is 8, and the loss function is the mean absolute error L1 loss function.
[0134] 2. Algorithm performance comparison
[0135] In order to evaluate the performance of the proposed method, this experiment was compared with some methods on five public datasets. Includes VDSR (from Vaish V, Adams A. The (new) standford light field archive), EDSR (from Kim J, Lee JK, Lee K M. Accurate image super-resolution using very deepconvolutional networks), resLF (from Zhang S, Lin Y, Sheng H. Residual networks for light field image super-resolution), LFSSR (from Yoon Y, Jeon HG, Yoo D,etal.Light field image super-resolution using convolutional neural network), LFSSR-ATO (from Jin J, Hou JH, Chen J, et al. Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistencyregularization), MEG-Net (from Wang YQ, Yang JG, Wang LG, et al. Light field image super-resolution using deformable convolution), LF-Inter (from Yoon Y,JeonH G,Yoo D,et al.Light field image super-resolution using convolutional neuralnetwork), DPT (from Wang SZ, Zhou TF, Lu Y, et al. Detail-preserving transformer for light field image super-resolution), where VDSR and EDSR directly use the corresponding SISR method to super-resolve each view in the light field image.
[0136] The indicators for evaluating super-resolution performance are Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM). Table 1 and Table 2 are comparisons of double super-resolution results and quadruple super-resolution results, respectively. The bold ones are the best results, and the underlined ones are the suboptimal results. It can be seen from the results in the table that the indicators of the method of the present invention are all optimal or suboptimal, proving that the method of the present invention has good super-resolution performance.
[0137] Table 1 Comparison of PSNR (dB) / SSIM of different methods ×2 super-resolution
[0138]
[0139] Table 2 PSNR (dB) / SSIM comparison of different ×4 super-resolution methods
[0140]
[0141] As can be seen from Table 1, in the two-fold super-resolution results, the method proposed in the present invention has achieved good performance on all five databases. Among them, the PSNR index is the best on all five databases, and the SSIM index is the best on the remaining three databases except for the EPFL and INRIA_Lytro databases. Compared with other light field image SR methods, VDSR and EDSR directly use the SISR method to super-resolve each view in the light field, ignoring the correlation between different views, resulting in a large gap between the super-resolved results and other methods. Since both MEG-Net and the method of the present invention use EPI in the SR process, and EPI can effectively reflect the scene geometry consistency of the light field, the SSIM index is higher than other methods. In the four-fold super-resolution results, the PSNR index of the method of the present invention is suboptimal on the EPFL database, and is the best on the remaining four databases, and the SSIM index is the best on the five databases. It can be proved that the method of the present invention can maintain a high super-resolution quality under different multiples of super-resolution tasks.
[0142] Figure 7 and Figure 8 The left side of the figure shows the central view of the corresponding scene, and the right side shows the enlarged area after the central view of the SR of different methods. The PSNR and SSIM of the corresponding methods in this scene are listed below. Figure 7 The Building_Decoded scene from the selected INRIA_Lytro database, Figure 8 This is the Origami scene in the HCI_new database. Figure 7 and Figure 8It can be seen from Figure 1 that the visual effect of directly applying the single image SR algorithm to the light field image is obviously not as good as the light field image SR algorithm. Figure 7 It can be seen that the proposed method is closer to the original light field image. At the black texture at the yellow arrow, the SR result of the proposed method is clearer and can restore more texture details compared with other methods. The EDSR method does not restore the texture details, and the image is blurred as a whole. Although the other four light field image SR algorithms restore some texture details, the restored texture details are not clear enough. Figure 8 It can be seen that at the black stripes of the yellow arrow, although LF-Inter, DPT and MEG methods all reconstructed the black stripes, the black stripes reconstructed by the method of the present invention are clearer. In addition, the reconstruction effect of the other methods is poor at the edge details at the green arrow. Since the method of the present invention integrates the gradient multiple sub-pixel information during the reconstruction process, the reconstruction result restores the edge texture details of the light field image.
[0143] 3. Computational efficiency analysis
[0144] In order to further verify the computational efficiency of the method of the present invention, this experiment evaluates different methods from the perspective of the number of parameters of the network model and the computational complexity Flops. Table 3 shows the number of parameters, Flops, and average PSNR and SSIM on five databases of different super-resolution models. When calculating Flops, the size of the input light field image is set to 5×5×32×32. It can be seen from Table 3 that the model of the method of the present invention has the highest average PSNR and SSIM values when the number of parameters and Flops are small, and has a higher reconstruction quality. Compared with EDSR and LFSSR-ATO, the Flops of the method of the present invention is significantly lower. In order to protect the edge details of the image during reconstruction, the method of the present invention designs a gradient branch, so the number of parameters and Flops are slightly higher than other methods.
[0145] Table 3×2 super-resolution model parameter quantity, Flop and PSNR / SSIM comparison
[0146]
[0147] 4. Ablation experiment
[0148] In order to explore the effectiveness of sub-pixel information and gradients in the process of light field image super-resolution, the present invention conducted four sets of ablation experiments. The experiment was verified on the EPFL database, where 70 scenes were used for training and 10 scenes were used for testing. The evaluation indicators are PSNR and SSIM. The experimental results are shown in Table 4. The feature extraction part used for comparison is the same as that disclosed in "Wang YQ, Yang J G, Wang LG, et al. Light field image super-resolution using deformable convolution" and "Wang SZ, Zhou TF, Lu Y, et al. Detail-preserving transformer for light field image super-resolution", which consists of two residual atrous spatial pyramid pooling blocks (Residual Atrous Spatial Pyramid Pooling Blocks, ResASPP) and residual blocks (ResidualBlocks, RB). MSPIE is a multiple sub-pixel information extraction module proposed in the present invention.
[0149] Table 4. Comparison of ablation experiments
[0150]
[0151] The first and third groups of experiments in Table 4 are comparisons between the ResASPP+RB structure and the MSPIE module, respectively. The second and fourth groups of experiments are comparisons after adding gradient details, respectively.
[0152] The experimental results show that compared with the ResASPP+RB structure, the MSPIE module of the method of the present invention has improved PSNR and SSIM indicators with a small increase in the number of parameters and Flops, which proves the effectiveness of the MSPIE module in the feature extraction part. By using the sub-pixel information of the light field image, the reconstruction quality can be effectively improved. Secondly, the role of gradient details is verified. The experimental results show that the reconstruction quality of adding gradient details is higher than that of not having gradient details, which proves that the fusion of gradients in the reconstruction process can effectively protect the edge texture details of the light field image, thereby improving the reconstruction quality.
[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.
Claims
1. Light field image super-resolution reconstruction method based on sub-pixel and gradient guidance, It is characterized in that include: S1: Acquire the target light field image to be reconstructed; S2: Input the target light field image into the trained super-resolution reconstruction model and output the corresponding super-resolution reconstructed image; The super-resolution reconstruction model is trained through the following steps: S201: first, converting a high-resolution light field image used as a sample into a low-resolution light field image; then extracting a gradient map of the low-resolution light field image to obtain a corresponding gradient array; finally, using the low-resolution light field image and the corresponding gradient array as model input; S202: extracting the low-resolution light field image and the multiple sub-pixel features of the corresponding gradient array through a multiple sub-pixel information extraction module to obtain corresponding light field multiple sub-pixel features and gradient multiple sub-pixel features; S203: performing feature representation enhancement on the light field multiple sub-pixel feature and the gradient multiple sub-pixel feature through an enhancement module to obtain corresponding light field multiple sub-pixel enhancement features and gradient multiple sub-pixel enhancement features; S204: performing feature fusion on the light field multiple sub-pixel enhancement feature and the gradient multiple sub-pixel enhancement feature through a fusion module to obtain a corresponding multiple sub-pixel fusion feature; S205: performing super-resolution reconstruction based on multiple sub-pixel fusion features through a reconstruction module to obtain a reconstructed image; then adding the low-resolution light field image to the reconstructed image after bicubic interpolation to generate a corresponding super-resolution reconstructed image; S206: calculating a training loss according to the super-resolution reconstructed image and the high-resolution light field image used as a sample, and optimizing parameters of the super-resolution reconstruction model based on the training loss; S207: repeating steps S201 to S206 until the super-resolution reconstruction model converges; S3: Using the output super-resolution reconstructed image as the super-resolution reconstruction result of the target light field image.
2. The light field image super-resolution reconstruction method based on sub-pixel and gradient guidance as claimed in claim 1, Features: In step S202, the low-resolution light field image or gradient array is stacked into four image piles in the horizontal, vertical, diagonal and anti-diagonal directions and input into the multiple sub-pixel information extraction module: first, the sub-pixel feature information of the four image piles is extracted respectively; then the sub-pixel feature information of each image pile is connected to obtain the global sub-pixel feature; finally, the global sub-pixel feature is reduced in dimension and aligned in feature to obtain the corresponding multiple sub-pixel features.
3. The light field image super-resolution reconstruction method based on sub-pixel and gradient guidance as claimed in claim 2, Features: The multiple sub-pixel information extraction module includes four feature multiple sub-pixel information extraction modules that take four image stacks as input and are used to extract sub-pixel feature information, a connection module that is used to connect the sub-pixel feature information output by the four feature multiple sub-pixel information extraction modules, a 3D convolution module that is used to reduce the dimension of the connected global sub-pixel features, and an angle alignment module that is used to align the features of the reduced global sub-pixel features; the output of the angle alignment module is the corresponding multiple sub-pixel features.
4. The light field image super-resolution reconstruction method based on sub-pixel and gradient guidance as claimed in claim 3, Features: The feature multiple sub-pixel information extraction module includes a 3D convolution layer and four stacked residual blocks connected end to end in sequence, and the input of the first residual block and the output of the last residual block are summed as the sub-pixel feature information output by the feature multiple sub-pixel information extraction module; each residual block includes a PReLu activation layer and a 3D convolution layer connected end to end in sequence, and the input of the PReLu activation layer and the output of the 3D convolution layer are summed.
5. The light field image super-resolution reconstruction method based on sub-pixel and gradient guidance as claimed in claim 1, Features: In step S203, the light field multiple sub-pixel features or the gradient multiple sub-pixel features are converted into corresponding feature sequences and then input into the enhancement module: first, the sub-pixel spatial features of the feature sequence are extracted; then, the corresponding query matrix, keyword matrix and value matrix are generated based on the sub-pixel spatial features, and a spatial feature map is generated according to the three matrices; finally, the spatial feature map is added to the input feature sequence to obtain the corresponding multiple sub-pixel enhanced features.
6. The light field image super-resolution reconstruction method based on sub-pixel and gradient guidance as claimed in claim 5, It is characterized in that The enhancement module includes the following processing steps: S2031: extracting sub-pixel spatial features of the feature sequence respectively through three convolutional layer branches; S2032: Expand the three sub-pixel spatial features respectively to obtain corresponding local feature blocks; then reshape the three local feature blocks into corresponding query matrices Q 1 , keyword matrix K 1 Sum value matrix V 1 ; S2033: To K 1 Perform the transpose operation to obtain the keyword matrix transpose K 1 T ; S2034: K 1 T With Q 1 Multiply and add to V 1 Multiply, and then perform folding and convolution operations on the product to obtain the spatial feature map; S2035: Add the spatial feature map to the input feature sequence to obtain corresponding multiple sub-pixel enhanced features.
7. The light field image super-resolution reconstruction method based on sub-pixel and gradient guidance as claimed in claim 1, Features: In step S204, the light field multiple sub-pixel enhancement feature and the gradient multiple sub-pixel enhancement feature are input into a fusion module: first, light field enhancement spatial features and gradient enhancement spatial features of the light field multiple sub-pixel enhancement feature and the gradient multiple sub-pixel enhancement feature are extracted; Then, the corresponding query matrix, keyword matrix and value matrix are generated based on the light field enhanced spatial features and the gradient enhanced spatial features, and an enhanced feature map is generated according to the three matrices; finally, the enhanced feature map is added to the input light field multiple sub-pixel enhanced features to obtain the corresponding multiple sub-pixel fusion features.
8. The light field image super-resolution reconstruction method based on sub-pixel and gradient guidance as claimed in claim 7, It is characterized in that The fusion module includes the following processing steps: S2041: Extracting light field enhancement spatial features of multiple sub-pixel enhancement features of the light field through a convolutional layer branch; S2042: respectively extracting gradient enhanced spatial features of gradient multiple sub-pixel enhanced features through two convolutional layer branches; S2043: Expand the light field enhanced spatial features to obtain a corresponding local feature block; then reshape the local feature block into a corresponding query matrix Q 2 ; S2044: Expand the two gradient enhanced spatial features to obtain corresponding local feature blocks; then reshape the two local feature blocks into corresponding keyword matrices K 2 Sum value matrix V 2 ; S2045: To K 2 Perform the transpose operation to obtain the keyword matrix transpose K 2 T ; S2046: K 2 T With Q 2 After multiplication, V 2 Multiply, and then perform folding and convolution operations on the resulting product to obtain an enhanced feature map; S2047: Add the enhanced feature map to the input light field multiple sub-pixel enhanced feature to obtain a corresponding multiple sub-pixel fusion feature.
9. The light field image super-resolution reconstruction method based on sub-pixel and gradient guidance as claimed in claim 1, Features: The reconstruction module consists of an information multi-distillation module and an upsampling module, where the upsampling module contains two convolutional layers and a pixel reorganization layer; When the reconstruction module performs super-resolution reconstruction: first, the multi-level feature information is extracted from the multiple sub-pixel fusion features through the information multi-distillation module; then the channel dimension of the multi-level feature information is increased to a through the first convolutional layer. 2 N, and the corresponding reconstructed features are obtained; then the reconstructed features are upsampled to the target resolution aH×aW through the pixel recombination layer; finally, the channel dimension of the upsampled reconstructed features is reduced to 1 through the second convolutional layer to obtain the corresponding reconstructed image.
10. The light field image super-resolution reconstruction method based on sub-pixel and gradient guidance as claimed in claim 1, It is characterized in that In step S206, the training loss is calculated using the following formula: Where: loss(x,y) represents the training loss; f(x i ) represents the generated super-resolution reconstructed image, x i represents the input low-resolution light field image; y i represents the high-resolution light field image used as a sample; n represents the number of samples.
