Dense light field reconstruction method based on space-angle multi-dimensional interaction and guide residual network

Abstract

The invention provides an efficient end-to-end sparse-to-dense light field reconstruction method based on space-angle multi-dimensional interaction and a guide residual network, and the method comprises the steps: enabling a sparse view collected by a light field camera to serve as the input of a network, and generating a dense light field sub-view through a designed space-angle multi-dimensional interaction module; and performing view optimization on the generated dense light field sub-view through a guide residual refining module, and finally realizing dense light field reconstruction. The method comprises the specific steps that firstly, an input sparse light field view is converted into a YCrCb format from an RGB format; secondly, space-angle information of a Y channel view is extracted; then a space-angle multi-dimensional interaction module generates dense sub-views according to information extracted from the Y channel view; performing texture detail optimization on the generated dense sub-views through a guide residual refining module; and finally, orderly arranging the optimized dense sub-views and the original view to realize dense light field reconstruction.

Description

Technical Field

[0001] This invention proposes an efficient end-to-end method for sparse-to-dense light field reconstruction based on spatial-angle multi-dimensional interaction and guided residual networks. The method uses a sparse view captured by a light field camera as the input view for the network. A designed spatial-angle multi-dimensional interaction module generates a dense light field sub-view. The generated dense light field sub-view is further optimized using a guided residual refinement module. Finally, the optimized sub-view is compared with the original input view according to their angular positions. The orderly arrangement enables the reconstruction of a dense light field. Background Technology

[0002] Traditional two-dimensional planar cameras only record the intensity and position of light rays, with each pixel capturing light rays from only a single direction, thus losing information such as the direction and angle of the light rays. Light field cameras, on the other hand, record information related to the direction and intensity of light rays passing through the lens, thereby effectively preserving the three-dimensional information of the scene. The stored light field information can be applied to fields such as computer vision and light field imaging, for example, refocusing after light field capture, light field view synthesis, and depth estimation.

[0003] The device used in a light field camera to acquire light field images mainly consists of a microlens array placed between the main lens and the image sensor to record the direction and intensity information of incident light rays. This overcomes the physical focusing limitations of traditional cameras, allowing for post-capture focusing and recording of four-dimensional light data—two-dimensional spatial and two-dimensional angular information—to achieve light field acquisition. However, the physical size and resolution of the image sensor in a light field camera are limited, necessitating a trade-off between spatial and angular resolution of the light field image. For example, dense angular domain sampling inevitably leads to sparse spatial domain sampling; conversely, sparse angular domain sampling achieves higher spatial resolution, displaying more detailed spatial textures and details, but at the cost of a reduced visible angle of view. Therefore, to improve the quality of generated light field images, it is necessary to reconstruct the light field from sparse to dense.

[0004] To address the trade-off between spatial and angular resolution in light field image acquisition, a growing number of scholars and experts have begun researching super-resolution reconstruction of light field images in recent years. Within this field, there are primarily traditional methods and deep learning-based methods. Traditional methods typically improve resolution by designing filters and depth estimation; however, they fail to fully utilize the complementary information contained in the viewpoint, resulting in generally mediocre super-resolution performance. Deep learning-based methods outperform traditional methods in light field image super-resolution reconstruction. To address the inherent trade-off between angular and spatial resolution, this paper proposes a method that reconstructs the image into two parts: disparity and color estimation. These parts are then modeled separately using a sequential convolutional neural network, and the network is trained by minimizing a loss function. An improved one-step method for achieving spatial and angular super-resolution of the light field adaptively aggregates effective pixels from different sub-aperture images in the input light field and incorporates feature separation and interaction modules to refine the details of the reconstructed light field sub-views. A method based on normalization of a set of sub-views in the feature map, designing a staged loss function, and employing a multi-range training strategy improves the quality of the reconstructed image. Finally, a method using a blur-deblurring scheme is proposed to address this issue. There is an information asymmetry problem between the spatial and angular domains. Convolutional neural networks are used to map the blurred epipolar plane from low resolution to high resolution. The spatial resolution of sub-aperture images is enhanced by a spatial super-resolution network. Then, three different angular super-resolution networks are used to generate new views between sub-aperture images based on the position of the new viewpoint. Finally, weight sharing technology is used to improve training quality and achieve dense light field reconstruction. A spatial-angular multi-scale super-resolution decoupling network is proposed to reconstruct high-resolution light field images. First, the correspondence between sub-pixel information of different parallax ranges between sub-aperture images in the light field is established. Then, the sub-pixel information of each dimension is fused to generate global high-frequency details of the image. Finally, the upsampled low-frequency and high-frequency details are combined to generate high-resolution light field images. Summary of the Invention

[0005] The purpose of this invention is to address the problem that, during the acquisition of light field images, the limited physical size and resolution of the image sensor in a light field camera result in a sparse light field view that cannot reflect the complete information of the entire scene or the stereoscopic depth perception information of the light field. This invention proposes a spatial-angle multi-dimensional interaction module and designs a guided residual refinement network to reconstruct a dense light field from the input sparse light field. To achieve the above objective, the steps of this invention are as follows:

[0006] Step 1: Format conversion of the input sparse light field view. Convert the input sparse light field view from RGB format to YCbCr format, and use the view information of the Y channel as the input to the network model for generating dense light field subviews.

[0007] Step 2: Extraction of Spatial-Angular Structure Information. Spatial and angular domain information is extracted from the Y-channel view of the input network. This extracted information is used as valid input for subsequent network models.

[0008] Step 3: The spatial-angle multi-dimensional information interaction module generates a dense light field subview. The effective information from the spatial and angular domains of the Y-channel view obtained in Step 2 is input into the module for feature learning. Channel attention is added to the module to generate a feature-enhanced dense subview. The number of spatial-angle multi-dimensional interaction modules depends on the learning depth; the greater the depth, the longer the training time and the more detailed the learned feature information. Finally, by aligning and stitching with the input sparse view, a complete dense view is generated as the output.

[0009] Step 4: Guide the residual refinement module to optimize the generated dense subviews. Input the dense subviews generated in Step 3 into the guided residual refinement module for view refinement, thereby learning the high-frequency feature information of the generated dense views to improve image quality.

[0010] Step 5: Arrange the optimized dense subview and the original input view in order according to the angular position (s,t) to realize the reconstruction from sparse light field view to dense light field view.

[0011] The specific details of step one are as follows:

[0012] The original RGB format of the sparse light field view is converted to YCbCr format. The human eye is more sensitive to luminance (Y) and less sensitive to color (CbCr). Dense light field is generated using information from the Y channel view. The formula for converting RGB format to YCbCr is as follows.

[0013]

[0014]

[0015]

[0016] The specific details of step two are as follows:

[0017] The sparse light field view in YCbCr format obtained in step one is decomposed into luminance channel Y, chrominance channel Cb and Cr components. Then, 4D structural information is extracted from the sparse view of the luminance channel Y component, which are the spatial dimension (uv) and angular dimension (st). Information extraction of the sparse light field view is achieved through convolution operation.

[0018] The specific details of step three are as follows:

[0019] (1) The spatial-angle multi-dimensional interaction module generates a dense light field view by performing spatial and angular domain convolution and information interaction on the information carried by the Y channel of the sparse light field view. The generation of the dense light field view includes the original input sparse light field view, and the generated light field subviews and sparse views are arranged according to their angular domain positions. The views are arranged in an ordered manner to fully utilize the available information between them. The Y component of the input sparse view is denoted as... The Y component of the generated intermediate dense view is denoted as The multi-dimensional interactive module from a spatial perspective is denoted as Then the function implemented by the module can be expressed as the formula.

[0020] The computational complexity using full 4D convolution is... The calculation formula is shown below. The computational complexity of pseudo-4D convolution (space-angle) convolution is... The calculation formula is shown below. Therefore, to reduce computational complexity, this invention uses pseudo-4D convolution for calculation. Pseudo-4D convolution refers to decomposing the complete 4D domain into two branches: the spatial domain and the angular domain. The convolution kernel size in the spatial domain and the angular domain is 3×3, and the stride is 1. Spatial-angular interaction refers to mapping the information extracted during the convolution process to each other, fully sharing information, thereby effectively capturing cross-view consistency and spatial texture while reducing computational complexity.

[0021]

[0022]

[0023] in, This represents the computational complexity of a complete 4D convolution. This represents the computational complexity of pseudo-4D convolution. The size of the convolution kernel represents the spatial dimension. This indicates the size of the convolution kernel in the angular dimension. Indicates the number of input channels. Indicates the number of output channels.

[0024] (2) The channel attention mechanism (CA) at the end of the module strengthens the expression of effective information, suppresses redundant information and noise, and improves the quality of the generated light field subview by performing average pooling and adaptive learning of channel weights on the fused information.

[0025] The specific details of step four are as follows:

[0026] (1) After passing through the space-angle multi-dimensional interaction module stack, the original sparse subview is used to generate a dense view, and the sparse view and the generated view are arranged according to their angular positions. Arranged in an orderly manner, we obtain a light field view with denser brightness in the middle, denoted as... However, the high-frequency information in the generated dense view is blurry and unclear. To further improve the quality of the dense view, a guided residual refinement module was designed. First, the generated... Input the network, then add the sparse view components. ,generate Sobel operator and As a guiding factor, it is spliced ​​in the channel dimension and fed into the subsequent network.

[0027] (2) The information of the guiding factor in (1) after being spliced ​​in the channel is downsampled twice in the angle domain. The size of the convolution kernel is 2×2, the stride is 2, and the Leaky Relu function is used with an activation parameter of 0.2.

[0028] (3) Angle aggregation: learnable fusion of angle domains to obtain condensed multi-view features.

[0029] (4) Perform residual mapping on the fused information for each channel to obtain the desired result. View residuals .

[0030] (5) Finally, According to the predefined angle position Arranged in an orderly manner, in the corresponding positions with Summing element-wise yields the intermediate output view of the guided residual refinement network, denoted as . .in, The calculation formula is shown below.

[0031]

[0032] (6) Finally, the sparse view components are... Restored to its original position using a predefined viewpoint index, this yields a dense light field view refined by guided residuals, denoted as... This significantly improves the quality and consistency of dense light field views while keeping the true values ​​of the input sparse view unchanged.

[0033] The specific details of step five are as follows:

[0034] (1) Check the dense light field subview obtained in step four. Determine the corresponding subview Location ( , ).

[0035] (2) The dense light field subview after inspection Arrange the original input view in order of angular position (s,t), and remember it. .

[0036] (3) By performing angular linear upsampling on the original sparse light field view of the CbCr two-channel view, the same number of dense light field views are interpolated and reconstructed, denoted as... .

[0037] (4) Combine (2) and (3) and Channel-wise fusion is performed to achieve the final dense optical field reconstruction in YCbCr format, denoted as... .

[0038] (5) Reconstruct the dense light field in (4) Convert to RGB format, denoted as This completes the reconstruction of the light field from sparse to dense.

[0039] The present invention also includes the following features:

[0040] Light field cameras are limited by high hardware requirements and large storage costs. Current practical acquisitions often only yield sparse light field images. Sparse light field views only retain limited viewpoint images, resulting in insufficient angular information. Furthermore, they are difficult to analyze scene information in occluded scenes. To address these issues, this invention designs a spatial-angle multi-dimensional interaction module and a guided residual refinement module to reconstruct sparse light fields into dense light fields, thereby solving the defects of insufficient scene information and limited observation angles.

[0041] Compared with other methods for dense light field reconstruction, the dense light field reconstruction method based on spatial-angle multidimensional interaction and guided residual network designed in this invention has a strong generalization ability. The PSNR / SSIM of the reconstructed light field subview indexes are superior to other methods, which is helpful for 3D reconstruction, depth estimation and practical engineering needs. Attached Figure Description

[0042] Figure 1 This is a flowchart of the method of the present invention.

[0043] Figure 2 Input-output relationship diagram for reconstructing the light field

[0044] Figure 3 A two-dimensional array of spatial-angular 4D light field sub-images Detailed Implementation

[0045] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings. The invention will now be further described in conjunction with the accompanying drawings and implementation steps.

[0046] This invention proposes an efficient end-to-end method for sparse-to-dense light field reconstruction based on spatial-angle multi-dimensional interaction and guided residual networks. The method uses a sparse view captured by a light field camera as the input view for the network. A designed spatial-angle multi-dimensional interaction module generates a dense light field sub-view. The generated dense light field sub-view is further optimized using a guided residual refinement module. Finally, the optimized sub-view is compared with the original input view according to their angular positions. The orderly arrangement enables the reconstruction of a dense light field.

[0047] Figure 1 The flowchart of the method of this invention is as follows: First, the input sparse light field view format is converted from RGB format to YCrCb format; second, the spatial-angular structure information of the Y channel view is extracted; then, the spatial-angular multi-dimensional interaction module generates a dense light field subview based on the spatial-angular structure information extracted from the Y channel view; third, the generated dense light field subview is optimized for texture details by guiding the residual refinement module; finally, the optimized dense light field subview is arranged in an orderly manner with the original light field view to achieve dense light field reconstruction.

[0048] The specific implementation steps are as follows:

[0049] Step 1 converts the original RGB format of the sparse light field view to YCbCr format. The human eye is more sensitive to brightness (Y) and less sensitive to color (CbCr). Dense light field is generated using information from the Y channel view. The formula for converting RGB format to YCbCr is as follows.

[0050]

[0051]

[0052]

[0053] Step 2 decomposes the sparse light field view in YCbCr format obtained in Step 1 into luminance channel Y, chrominance channel Cb and Cr components. Then, the 4D structural information of the sparse view of the luminance channel Y component is extracted, namely the spatial dimension (uv) and the angular dimension (st). The information extraction of the sparse light field view is achieved by convolution operation.

[0054] Step 3.1 The spatial-angular multi-dimensional interaction module generates a dense light field view by performing spatial and angular domain convolution and information interaction on the information carried by the Y channel of the sparse light field view. The generation of the dense light field view includes the original input sparse light field view, and the generated light field sub-views and sparse views are arranged according to their angular domain positions. The views are arranged in an ordered manner to fully utilize the available information between them. The Y component of the input sparse view is denoted as... The Y component of the generated intermediate dense view is denoted as The multi-dimensional interactive module from a spatial perspective is denoted as Then the function implemented by the module can be expressed as the formula.

[0055] The computational complexity using full 4D convolution is... The calculation formula is shown below. The computational complexity of pseudo-4D convolution (space-angle) convolution is... The calculation formula is shown below. Therefore, to reduce computational complexity, this invention uses pseudo-4D convolution for calculation. Pseudo-4D convolution refers to decomposing the complete 4D domain into two branches: the spatial domain and the angular domain. The convolution kernel size in the spatial domain and the angular domain is 3×3, and the stride is 1. Spatial-angular interaction refers to mapping the information extracted during the convolution process to each other, fully sharing information, thereby effectively capturing cross-view consistency and spatial texture while reducing computational complexity.

[0056]

[0057] in, This represents the computational complexity of a complete 4D convolution. This represents the computational complexity of pseudo-4D convolution. The size of the convolution kernel represents the spatial dimension. This indicates the size of the convolution kernel in the angular dimension. Indicates the number of input channels. Indicates the number of output channels.

[0058] Step 3.2 The channel attention mechanism (CA) at the end of the module enhances the expression of effective information, suppresses redundant information and noise, and improves the quality of the generated light field subview by performing average pooling and adaptive learning of channel weights on the fused information.

[0059] Step 4.1 After passing through the spatial-angle multi-dimensional interaction module stack, a dense view is generated from the original sparse subviews. The sparse view and the generated view are then compared according to their angular positions. Arranged in an orderly manner, we obtain a light field view with denser brightness in the middle, denoted as... However, the high-frequency information in the generated dense view is blurry and unclear. To further improve the quality of the dense view, a guided residual refinement module was designed. First, the generated... Input the network, then add the sparse view components. ,generate Sobel operator and As a guiding factor, it is spliced ​​in the channel dimension and fed into the subsequent network.

[0060] Step 4.2 The information of the guiding factor in Step 4.1 after concatenation in the channels is downsampled twice in the angular domain. The size of the convolution kernel is 2×2, the stride is 2, and the Leaky ReLU function is used with an activation parameter of 0.2.

[0061] Step 4.3 Angle aggregation: The angle domains are learned to be fused to obtain condensed multi-view features.

[0062] Step 4.4 Perform residual mapping on the fused information for each channel to obtain the desired result. View residuals .

[0063] Step 4.5 Finally, According to the predefined angle position Arranged in an orderly manner, in the corresponding positions with Summing element-wise yields the intermediate output view of the guided residual refinement network, denoted as . .in, The calculation formula is shown below.

[0064]

[0065] Step 4.6 Finally, combine the sparse view components. Restored to its original position using a predefined viewpoint index, this yields a dense light field view refined by guided residuals, denoted as... This significantly improves the quality and consistency of dense light field views while keeping the true values ​​of the input sparse view unchanged.

[0066] Step 5.1 Check the dense light field subview obtained in Step 4. Determine the corresponding subview Location ( , ).

[0067] Step 5.2 View the dense light field subview after inspection Arrange the original input view in order of angular position (s,t), and remember it. .

[0068] Step 5.3 The same number of dense light field views are reconstructed by interpolating the original sparse light field view's CbCr two-channel view through angular linear upsampling, denoted as... .

[0069] Step 5.4 Combine the steps in Step 5.2 and Step 5.3. and Channel-wise fusion is performed to achieve the final dense optical field reconstruction in YCbCr format, denoted as... .

[0070] Step 5.5 Reconstruct the dense light field from Step 5.4 Convert to RGB format, denoted as This completes the reconstruction of the light field from sparse to dense.

[0071] Figure 2 This is an input-output diagram for reconstructing the light field. It is a diagram for reconstructing an 8×8 dense light field from a 2×2 sparse light field. The four corner points with a white background are the original sparse light field view positions, and the 60 view positions with a light green background are the generated dense light field subviews. All 64 views constitute the reconstructed dense light field.

[0072] Figure 3 This is a two-dimensional array of 4D light field sub-images in space and angle. The number of views in the angle plane *st* is M×N. In the space plane *uv*, each sub-view contains W×H pixels. Thus, the two-dimensional array formed by the angle plane and the space plane can pinpoint any specific location.

Claims

1. A dense light field reconstruction method based on spatial-angle multi-dimensional interaction and guided residual network, characterized in that, Includes the following steps: Step 1: Convert the format of the input sparse light field view. Convert the RGB format of the input sparse light field view to YCbCr format, and use the view information of the Y channel as the input of the network model to generate dense light field subviews. Step 2: Extraction of spatial-angular structure information. Extract spatial and angular information from the Y-channel view of the input network and use the extracted information as effective input information for the subsequent network model. Step 3: The spatial-angle multi-dimensional information interaction module generates a dense light field subview. The effective information of the spatial domain and angle domain of the Y-channel view obtained in Step 2 is input into the module for feature learning. Channel attention is added to the module to generate a feature-enhanced dense subview. The number of spatial-angle multi-dimensional interaction modules is determined by the learning depth. The greater the depth, the longer the training time and the more detailed the learned feature information. Finally, by aligning and stitching with the input sparse view, a complete dense view is generated as output. Step 4: The guide residual refinement module optimizes the generated dense subviews. The dense subviews generated in Step 3 are input into the guide residual refinement module for view refinement, thereby learning the high-frequency feature information of the generated dense views to improve image quality. Step 5: Arrange the optimized dense subview and the original input view in order according to the angular position (s,t) to realize the reconstruction from sparse light field view to dense light field view.

2. The dense light field reconstruction method based on spatial-angle multi-dimensional interaction and guided residual network according to claim 1, characterized in that, The specific process in Step 1 is as follows: Step 1 converts the original RGB format of the sparse light field view to YCbCr format. The human eye is more sensitive to brightness (Y) and less sensitive to color (CbCr). Dense light field is generated using information from the Y channel view. The formula for converting RGB format to YCbCr is as follows.

3. The dense light field reconstruction method based on spatial-angle multi-dimensional interaction and guided residual network according to claim 1, characterized in that, The specific process in Step 2 is as follows: The sparse light field view in YCbCr format obtained in Step 1 is decomposed into luminance channel Y, chrominance channel Cb and Cr components. Then, the 4D structural information of the sparse view of the luminance channel Y component is extracted, namely the spatial dimension (uv) and the angular dimension (st). The information extraction of the sparse light field view is achieved by convolution operation.

4. The dense light field reconstruction method based on spatial-angle multi-dimensional interaction and guided residual network according to claim 1, characterized in that, The specific process in Step 3 is as follows: Step 3.1 The spatial-angular multi-dimensional interaction module generates a dense light field view by performing spatial and angular domain convolution and information interaction on the information carried by the Y channel of the sparse light field view. The generation of the dense light field view includes the original input sparse light field view, and the generated light field sub-views and sparse views are arranged according to their angular domain positions. The views are arranged in an ordered manner to make full use of the available information between them. The Y component of the input sparse view is denoted as... The Y component of the generated intermediate dense view is denoted as The multi-dimensional interactive module from a spatial perspective is denoted as The function implemented by the module can be expressed as: The computational complexity using full 4D convolution is... The calculation formula is shown below. The computational complexity of pseudo-4D convolution (space-angle) convolution is... The calculation formula is shown in the following formula. Therefore, in order to reduce the computational complexity, this invention uses pseudo-4D convolution for calculation. Pseudo-4D convolution refers to decomposing the complete 4D domain into two branches, namely the spatial domain and the angular domain. The convolution kernel size in the spatial domain and the angular domain is 3×3, and the stride is 1. Spatial-angular interaction refers to mapping the information extracted during the convolution process to each other and fully sharing information, thereby effectively capturing cross-view consistency and spatial texture while reducing computational complexity. in, This represents the computational complexity of a complete 4D convolution. This represents the computational complexity of pseudo-4D convolution. The size of the convolution kernel represents the spatial dimension. This indicates the size of the convolution kernel in the angular dimension. Indicates the number of input channels. Indicates the number of output channels; Step 3.2 The channel attention mechanism (CA) at the end of the module enhances the expression of effective information, suppresses redundant information and noise, and improves the quality of the generated light field subview by performing average pooling and adaptive learning of channel weights on the fused information.

5. The dense light field reconstruction method based on spatial-angle multi-dimensional interaction and guided residual network according to claim 1, characterized in that, The specific process in Step 4 is as follows: Step 4.1 After passing through the spatial-angle multi-dimensional interaction module stack, a dense view is generated from the original sparse subviews. The sparse view and the generated view are then compared according to their angular positions. Arranged in an orderly manner, we obtain a light field view with denser brightness in the middle, denoted as... However, the high-frequency information in the generated dense view is blurry and unclear. To further improve the quality of the dense view, a guided residual refinement module was designed. First, the generated... Input the network, then add the sparse view components. ,generate Sobel operator and As a guiding factor, it is spliced ​​in the channel dimension and fed into the subsequent network; Step 4.2 The information of the guiding factor in Step 4.1 after concatenation in the channels is downsampled twice in the angular domain. The size of the convolution kernel is 2×2, the stride is 2, and the Leaky ReLU function is used with an activation parameter of 0.

2. Step 4.3 Angle aggregation: The angle domains are learned to be fused to obtain condensed multi-view features; Step 4.4 Perform residual mapping on the fused information for each channel to obtain the desired result. View residuals ; Step 4.5 Finally, According to the predefined angle position Arranged in an orderly manner, in the corresponding positions with Summing element-wise yields the intermediate output view of the guided residual refinement network, denoted as . ,in, The calculation formula is shown below; Step 4.6 Finally, combine the sparse view components. Restored to its original position using a predefined viewpoint index, this yields a dense light field view refined by guided residuals, denoted as... This significantly improves the quality and consistency of dense light field views while keeping the true value of the input sparse view unchanged.

6. The dense light field reconstruction method based on spatial-angle multi-dimensional interaction and guided residual network according to claim 1, characterized in that, The specific process in Step 5 is as follows: Step 5.1 Check the dense light field subview obtained in Step 4. Determine the corresponding subview Location ( , ); Step 5.2 View the dense light field subview after inspection Arrange the original input view in order of angular position (s,t), and remember it. ; Step 5.3 The same number of dense light field views are reconstructed by interpolating the original sparse light field view's CbCr two-channel view through angular linear upsampling, denoted as... ; Step 5.4 Combine the steps in Step 5.2 and Step 5.

3. and Channel-wise fusion is performed to achieve the final dense optical field reconstruction in YCbCr format, denoted as... ; Step 5.5 Reconstruct the dense light field from Step 5.4 Convert to RGB format, denoted as This completes the reconstruction of the light field from sparse to dense.

Images