Light field image space angle joint reconstruction method

Abstract

The invention relates to a light field image space angle joint reconstruction method. The method comprises the following steps: extracting Y channel information of a light field sub-aperture image array with low angle resolution and low space resolution; performing feature reconstruction on the extracted Y channel information to obtain Y channel information of the light field sub-aperture image array with high angular resolution and high spatial resolution; acquiring U channel information and V channel information of the light field sub-aperture array image with high angular resolution and high spatial resolution; and combining the Y channel information after feature reconstruction with the U and V channel information of the light field sub-aperture array image with the high angle resolution and the high spatial resolution, and converting the combined data into an RGB space to obtain an RGB image of the light field sub-aperture array image with the high angle resolution and the high spatial resolution. According to the method, high-quality light field sub-aperture array images with high angle resolution and high spatial resolution can be obtained, and the requirements of applications such as three-dimensional reconstruction, virtual reality and depth estimation are met.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a light field image spatial angle joint reconstruction method. Background Art

[0002] Light field imaging technology is an advanced imaging method that can capture the light distribution of an object in three-dimensional space, thereby providing more information than traditional two-dimensional imaging. Light field images with high angular resolution and high spatial resolution are widely used in depth estimation, biomedical testing, three-dimensional reconstruction, virtual reality and other fields. However, due to hardware limitations, it is impossible to obtain light fields with high angular resolution and high spatial resolution at one time, which has become the main obstacle to the development of light field imaging technology. In addition, most existing methods can only solve the problems of low spatial resolution or low angular resolution. Research on joint reconstruction of angle and space is still insufficient. Therefore, reconstructing light field images with high angular resolution and high spatial resolution from light field images with low angular resolution and low spatial resolution has important theoretical and practical application value. Summary of the invention

[0003] The purpose of the present invention is to provide a light field image spatial angle joint reconstruction method based on the light field image representation form of sub-aperture images to reconstruct light field images with high angular resolution and high spatial resolution, so as to solve the problems existing in the above-mentioned prior art.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A light field image spatial angle joint reconstruction method, comprising:

[0006] Convert the light field sub-aperture image array with low angular resolution and low spatial resolution from RGB space to YUV space and extract the Y channel information;

[0007] Reconstructing features of Y channel information of a light field sub-aperture image array with low angular resolution and low spatial resolution to obtain Y channel information of a light field sub-aperture image array with high angular resolution and high spatial resolution;

[0008] Performing bilinear upsampling processing on U channel information and V channel information of a light field sub-aperture array image with low angular resolution and low spatial resolution to obtain U channel information and V channel information of a light field sub-aperture array image with high angular resolution and high spatial resolution;

[0009] The Y channel information of the light field sub-aperture array with high angular resolution and high spatial resolution is merged with the U channel information and V channel information of the light field sub-aperture array image with high angular resolution and high spatial resolution, and the merged data is converted into RGB space to obtain an RGB image of the light field sub-aperture array image with high angular resolution and high spatial resolution.

[0010] Optionally, reconstructing features of Y channel information of a light field sub-aperture image array with low angular resolution and low spatial resolution includes:

[0011] Merging the Y channel information of the light field sub-aperture array with low angular resolution and low spatial resolution to obtain a four-dimensional light field sub-aperture array image with low angular resolution and low spatial resolution;

[0012] Using an angle feature reconstruction network, the angle feature of the four-dimensional light field sub-aperture array image with low angle resolution and low spatial resolution is reconstructed to obtain Y channel information of the light field sub-aperture array image with high angle resolution and low spatial resolution;

[0013] The spatial feature reconstruction network is used to reconstruct the Y channel information of the light field sub-aperture array image with high angular resolution and low spatial resolution to obtain the Y channel information of the light field sub-aperture array image with high angular resolution and high spatial resolution.

[0014] Optionally, the angle feature reconstruction network includes: a feature extraction module, a feature refinement module and a feature enhancement module;

[0015] The feature extraction module is used to extract the high-dimensional feature vector of the four-dimensional light field sub-aperture array image with low angular resolution and low spatial resolution;

[0016] The feature refinement module is used to rearrange the dimensions of the high-dimensional feature vector and perform feature refinement processing;

[0017] The feature enhancement module is used to enhance the high-dimensional feature vector after feature refinement.

[0018] Optionally, the feature extraction module includes: a first convolution layer, a LeakyReLU excitation layer, a first 3D convolution module, a second convolution layer, a second 3D convolution module, a third convolution layer, a third 3D convolution module, a fourth convolution layer, a fourth 3D convolution module, a fifth convolution layer, a fifth 3D convolution module, a sixth convolution layer, a seventh convolution layer, a first Summation layer, a second Summation layer and a Concatenation layer connected in sequence; wherein the 3D convolution module includes: a first 3D convolution layer, a LeakyReLU excitation layer, a second 3D convolution layer and a Summation layer connected in sequence.

[0019] Optionally, the feature refinement module includes: four first DE convolution modules connected in sequence; wherein the first DE convolution module includes: DEConv, Conv_1, Conv_2, 2 Summation layers and 1 LeakyReLU excitation layer;

[0020] The features input into the first DE convolution module first pass through DEConv, and then are summed with the input features through the Summation layer; the summed features pass through Conv_1 and the input features through the Summation layer, and finally pass through Conv_2 to obtain the output of the first DE convolution module.

[0021] Optionally, the feature enhancement module includes: EN_1, EN_2, LeakyReLU, EN_3, PixelShuffle, EN_4, LeakyReLU and 1 local implicit function;

[0022] The features of the input feature enhancement module first pass through a local implicit function to obtain implicitly enhanced features, and then pass through EN_1, EN_2, LeakyReLU, EN_3, PixelShuffle, EN_4, and LeakyReLU in sequence to obtain enhanced features.

[0023] Optionally, the spatial feature reconstruction network includes: a first interaction module, a second interaction module, a third interaction module, a fourth interaction module, a fifth interaction module, a sixth interaction module, UP_1, a PixelShuffle layer, and UP_2 connected in sequence;

[0024] The features of the input spatial feature reconstruction network are sequentially passed through the first interaction module, the second interaction module, the third interaction module, the fourth interaction module, the fifth interaction module, the sixth interaction module, UP_1, the PixelShuffle layer, and UP_2 to obtain the Y channel information of the light field sub-aperture image array with high spatial resolution and high angular resolution;

[0025] Each interaction module includes: four second DE convolution modules connected in sequence; wherein the second DE convolution module includes: DEConv, Conv_1, Conv_2, 2 Summation layers and 1 LeakyReLU excitation layer;

[0026] The features input to the second DE convolution module first pass through DEConv, and then are summed with the input features through the Summation layer. The summed features pass through Conv_1 and the input features through the Summation layer, and finally pass through Conv_2 to obtain the output of the second DE convolution module.

[0027] The beneficial effects of the present invention are:

[0028] When light field cameras acquire images, the limitations of their sensors lead to a trade-off between spatial resolution and angular resolution: it is impossible to simultaneously capture light field images with high angular resolution and high spatial resolution. This limitation significantly restricts the application potential of light field imaging technology in multiple fields. To overcome this challenge, the present invention proposes an innovative method that can reconstruct light field images with high spatial resolution and high angular resolution using sparse light field sub-aperture images with low spatial resolution. Through this method, more detailed light field image sub-aperture images can be generated, which is expected to expand the application scope of light field imaging technology in various fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 It is a flow chart of a method for joint reconstruction of light field images in spatial angles according to an embodiment of the present invention;

[0031] Figure 2 This is a framework diagram of reconstructing a light field sub-aperture image array with an angular resolution of 7×7 and a spatial resolution of 64×64 using a light field sub-aperture image array with an angular resolution of 2×2 and a spatial resolution of 32×32 according to an embodiment of the present invention;

[0032] Figure 3 A feature extraction module according to an embodiment of the present invention;

[0033] Figure 4 A 3D convolution module according to an embodiment of the present invention;

[0034] Figure 5A feature refinement module according to an embodiment of the present invention;

[0035] Figure 6 A feature enhancement module according to an embodiment of the present invention;

[0036] Figure 7 A spatial feature reconstruction network according to an embodiment of the present invention;

[0037] Figure 8 An interactive module according to an embodiment of the present invention;

[0038] Fig. 9 It is the DE convolution module of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] like Figure 1 and Figure 2 As shown, this embodiment proposes a light field image spatial angle joint reconstruction method, including:

[0042] Convert the light field sub-aperture image array with low angular resolution and low spatial resolution from RGB space to YUV space and extract the Y channel information;

[0043] Reconstructing features of Y channel information of a light field sub-aperture image array with low angular resolution and low spatial resolution to obtain Y channel information of a light field sub-aperture image array with high angular resolution and high spatial resolution;

[0044] Performing bilinear upsampling processing on U channel information and V channel information of a light field sub-aperture array image with low angular resolution and low spatial resolution to obtain U channel information and V channel information of a light field sub-aperture array image with high angular resolution and high spatial resolution;

[0045] The Y channel information of the light field sub-aperture array with high angular resolution and high spatial resolution is merged with the U channel information and V channel information of the light field sub-aperture array image with high angular resolution and high spatial resolution, and the merged data is converted into RGB space to obtain an RGB image of the light field sub-aperture array image with high angular resolution and high spatial resolution.

[0046] The present embodiment discloses a method for joint reconstruction of light field images at spatial angles, including extracting Y channel information of light field sub-aperture array images with low angular resolution and low spatial resolution, building an angular feature reconstruction network and a spatial feature reconstruction network, bilinear upsampling of U and V channel information of light field sub-aperture array images with low angular resolution and low spatial resolution, and YUV space to RGB space conversion. The method of the present embodiment first extracts Y channel information from light field sub-aperture array images with low angular resolution and low spatial resolution; secondly, the Y channel information of light field sub-aperture array images with high angular resolution and low spatial resolution is reconstructed using the angular feature reconstruction network; then, the Y channel information of light field sub-aperture array images with high angular resolution and high spatial resolution is obtained through the spatial feature reconstruction network. Subsequently, the U and V channel information of light field sub-aperture array images with low angular resolution and low spatial resolution are bilinearly upsampled using a bilinear interpolation algorithm to obtain U and V channel information of light field sub-aperture array images with high angular resolution and high spatial resolution. Finally, the Y channel information of the final high angular resolution and high spatial resolution light field sub-aperture array image is merged with the U and V channel information of the high angular resolution and high spatial resolution light field sub-aperture array image, and the merged data is converted into RGB space to obtain the RGB image of the final high angular resolution and high spatial resolution light field sub-aperture array image. This embodiment can obtain high-quality high angular resolution and high spatial resolution light field sub-aperture array images, which can meet the needs of applications such as three-dimensional reconstruction, virtual reality, and depth estimation. In addition, this embodiment can reconstruct high angular resolution and high spatial resolution light field sub-aperture array images at one time, which can meet the real-time requirements of various applications.

[0047] Specifically, in this embodiment, converting the light field sub-aperture image array with low angular resolution and low spatial resolution from the RGB space to the YUV space, and extracting the Y channel information includes:

[0048] Extracting a light field subaperture array image L with low angular resolution and low spatial resolution from the light field image SSLR , whose dimensions are 3×x×y×h×w, where x×y is the light field subaperture array image L with low angular resolution and low spatial resolution SSLR The angular resolution of the neutron aperture image, h×w is the light field subaperture array image L with low angular resolution and low spatial resolution SSLR The spatial resolution of a single sub-aperture image in is , and 3 is the three RGB channels. First, the low angular resolution and low spatial resolution light field sub-aperture array image L SSLR Convert from RGB space to YUV space. Then extract the Y channel information Y of the YUV space SSLR, whose dimensions are 1×x×y×h×w. The details are as follows:

[0049] Convert the light field sub-aperture image array with low angular resolution and low spatial resolution from RGB space to YUV space, and extract the Y channel information Y of the light field sub-aperture image array SSLR In this embodiment, the dimensions of the light field sub-aperture image array with low angular resolution and low spatial resolution are 3×2×2×32×32, and the dimensions of the light field sub-aperture image array with high angular resolution and high spatial resolution after reconstruction are 3×7×7×64×64. The Y channel information Y of the light field sub-aperture image array with low angular resolution and low spatial resolution extracted is SSLR The dimensions are 1×2×2×32×32.

[0050] Further, feature reconstruction of the Y channel information of the light field sub-aperture image array with low angular resolution and low spatial resolution includes:

[0051] Merging the Y channel information of the light field sub-aperture array with low angular resolution and low spatial resolution to obtain a four-dimensional light field sub-aperture array image with low angular resolution and low spatial resolution;

[0052] The angular feature reconstruction network is used to reconstruct the angular features of the four-dimensional light field sub-aperture array image with low angular resolution and low spatial resolution, and the Y channel information of the light field sub-aperture array image with high angular resolution and low spatial resolution is obtained;

[0053] The spatial feature reconstruction network is used to reconstruct the Y channel information of the light field sub-aperture array image with high angular resolution and low spatial resolution, so as to obtain the Y channel information of the light field sub-aperture array image with high angular resolution and high spatial resolution.

[0054] Furthermore, the angle feature reconstruction network includes: a feature extraction module, a feature refinement module and a feature enhancement module;

[0055] A feature extraction module, used to extract high-dimensional feature vectors of a four-dimensional light field subaperture array image with low angular resolution and low spatial resolution;

[0056] Feature refinement module, used to rearrange the dimensions of high-dimensional feature vectors and perform feature refinement processing;

[0057] The feature enhancement module is used to enhance the high-dimensional feature vector after feature refinement.

[0058] Furthermore, the feature extraction module includes: a first convolution layer, a LeakyReLU excitation layer, a first 3D convolution module, a second convolution layer, a second 3D convolution module, a third convolution layer, a third 3D convolution module, a fourth convolution layer, a fourth 3D convolution module, a fifth convolution layer, a fifth 3D convolution module, a sixth convolution layer, a seventh convolution layer, a first Summation layer, a second Summation layer and a Concatenation layer connected in sequence; wherein the 3D convolution module includes: a first 3D convolution layer, a LeakyReLU excitation layer, a second 3D convolution layer and a Summation layer connected in sequence.

[0059] Furthermore, the feature refinement module includes: four first DE convolution modules connected in sequence; wherein the first DE convolution module includes: DEConv, Conv_1, Conv_2, 2 Summation layers and 1 LeakyReLU excitation layer;

[0060] The features input to the first DE convolution module first pass through DEConv, and then are summed with the input features through the Summation layer; the summed features pass through Conv_1 and the input features through the Summation layer, and finally pass through Conv_2 to obtain the output of the first DE convolution module.

[0061] Furthermore, the feature enhancement module includes: EN_1, EN_2, LeakyReLU, EN_3, PixelShuffle, EN_4, LeakyReLU and 1 local implicit function;

[0062] The features of the input feature enhancement module first pass through a local implicit function to obtain implicitly enhanced features, and then pass through EN_1, EN_2, LeakyReLU, EN_3, PixelShuffle, EN_4, and LeakyReLU in sequence to obtain enhanced features.

[0063] Further, the spatial feature reconstruction network includes: a first interaction module, a second interaction module, a third interaction module, a fourth interaction module, a fifth interaction module, a sixth interaction module, UP_1, a PixelShuffle layer, UP_2, which are connected in sequence;

[0064] The features of the input spatial feature reconstruction network are sequentially passed through the first interaction module, the second interaction module, the third interaction module, the fourth interaction module, the fifth interaction module, the sixth interaction module, UP_1, the PixelShuffle layer, and UP_2 to obtain the Y channel information of the light field sub-aperture image array with high spatial resolution and high angular resolution;

[0065] Each interaction module includes: four second DE convolution modules connected in sequence; wherein the second DE convolution module includes: DEConv, Conv_1, Conv_2, 2 Summation layers and 1 LeakyReLU excitation layer;

[0066] The features input to the second DE convolution module first pass through DEConv, and then are summed with the input features through the Summation layer. The summed features pass through Conv_1 and the input features through the Summation layer, and finally pass through Conv_2 to obtain the output of the second DE convolution module.

[0067] Specifically, in this embodiment, first, the low-angle resolution light field sub-aperture array image Y channel information Y SSLR The two angular dimensions 2×2 are merged to obtain a four-dimensional light field sub-aperture array image Γ with low angular resolution and low spatial resolution, whose dimensions are 1×4×32×32. The four-dimensional light field sub-aperture array image Γ with low angular resolution and low spatial resolution is input into the angular feature reconstruction network to obtain the Y channel information of the light field sub-aperture array image with high angular resolution and low spatial resolution. The angular feature reconstruction network consists of three modules, namely, feature extraction module, feature refinement module and feature enhancement module. The feature extraction module is as follows Figure 3 As shown in Figure 2, the feature extraction module contains 16 layers, namely FE_1, LeakyReLU, 3D convolution module 1, FE_2, 3D convolution module 2, FE_3, 3D convolution module 3, FE_4, 3D convolution module 4, FE_5, 3D convolution module 5, FE_6, FE_7 and 2 Summation layers and 1 Concatenation layer. The convolution kernel sizes of FE_1, FE_2, FE_3, FE_4, FE_5, FE_6, and FE_7 are 1×1, 3×3×3, 3×3×3, 3×3×3, 3×3×3, 3×3, and 1×1 respectively, with step sizes of 1×1, 1×1×1, 1×1×1, 1×1×1, 1×1×1, 1×1, and padding values ​​of 0×0, 1×1×1, 1×1×1, 1×1×1, 1×1×1, 1×1, and 0×0 respectively. The 3D convolution module is shown in Figure 5. Figure 4 As shown in the figure, the 3D convolution module contains 4 layers, namely 3DConv_1, LeakyReLU, 3DConv_2 and 1 Summation layer. The convolution kernels of 3DConv_1 and 3DConv_2 are 3×3×3, the stride is 1×1×1, and the padding is 1×1×1.

[0068] The dimension of the high-dimensional feature vector after feature extraction is 1×64×4×32×32. Then, the high-dimensional feature vector after feature extraction is sent to the feature refinement module. Figure 5 As shown in . First, the dimension of the high-dimensional feature vector after feature extraction is rearranged to 1×64×64×64. The feature refinement module contains 4 layers, namely DE convolution module 1, DE convolution module 2, DE convolution module 3, and DE convolution module 4. The DE convolution module has 6 layers, namely DEConv, Conv_1, Conv_2, 2 Summation layers, and 1 LeakyReLU excitation layer. The convolution kernel of the DEConv layer is as follows: Fig. 9 As shown in Figure 2. The convolution kernels of DEConv, Conv_1, and Conv_2 are 3×3, 3×3, 1×1, the step size is 1×1, 1×1, 1×1, and the padding is 1×1, 1×1, 0×0. The high-dimensional feature vector z after feature refinement * The dimension of is 1×64×64×64. Then, the high-dimensional feature vector after feature refinement is sent to the feature enhancement module. Figure 6 As shown in Figure 1, the feature enhancement module contains 8 layers, namely EN_1, EN2, LeakyReLU, EN_3, PixelShuffle, EN_4, LeakyReLU and 1 local implicit function. The convolution kernels of EN_1, EN2, EN_3 and EN_4 are 1×1, 2×2, 1×1 and 3×3 respectively, with step sizes of 1×1, 2×2, 1×1 and 1×1 respectively, and the filling values ​​are 0×0, 0×0, 0×0 and 1×1 respectively. The local implicit function converts the high-dimensional feature vector z after feature refinement into * As a potential code, by q Position encoding is performed to expand the coordinates to high-frequency coordinates, so as to better characterize high-frequency signals. The specific description is:

[0069]

[0070] Among them, ω1 is 2e, ω L 2e L , x is the coordinate x q L is a modifiable network hyperparameter, which is set to 8 in this paper. Then the encoded high-frequency coordinate signal ψ(x) is combined with the potential code z * Input the local implicit function to get the pixel value of the corresponding coordinate. By traversing all coordinates, the pixel values ​​corresponding to all coordinates are obtained, and the enhanced high-dimensional feature vector is generated. The specific description is:

[0071] M(x q ) = f θ (z * ,ψ(xq ))(2).

[0072] Among them, f θ (·) refers to the local latent function module, which is implemented by a multi-layer perceptron. M(·) is the latent code z according to the input * With coordinates ψ(x q ) The pixel values ​​predicted by PixelShuffle are upsampled to the target resolution. The high-dimensional feature vector after the angle feature enhancement network has a dimension of 1×49×64×32×32, which is the Y channel information of the light field subaperture array image with high angular resolution and low spatial resolution.

[0073] Secondly, the Y channel information of the light field sub-aperture array image with high angular resolution and low spatial resolution is sent to the spatial feature reconstruction network to obtain the Y channel information of the light field sub-aperture array image with high angular resolution and high spatial resolution, whose dimension is 1×7×7×64×64. The spatial feature reconstruction network is as follows Figure 7 As shown in FIG. 1 . The spatial feature reconstruction network comprises 9 layers, namely, interaction module 1, interaction module 2, interaction module 3, interaction module 4, interaction module 5, interaction module 6, UP_2 and a PixelShuffle layer. The interaction module is shown in FIG. Figure 8 As shown. The interaction module contains 9 layers, namely DE convolution module 1, DE convolution module 2, DE convolution module 3, DE convolution module 4, Conv_1. DE convolution module 1, DE convolution module 2, DE convolution module 3, DE convolution module 4 are as shown. Fig. 9 As shown. The convolution kernels of DEConv, Conv_1, and Conv_2 are 3×3, 3×3, 1×1, with step sizes of 1×1, 1×1, 1×1, and padding of 1×1, 1×1, and 0×0. The convolution kernel size used by Conv_1 is 3×3, with a step size of 1×1, and a padding value of 0×0. The convolution kernel sizes used by UP_1 and UP_2 are 1×1 and 3×3, with step sizes of 1×1 and 1×1, and padding values ​​of 0×0 and 1×1. The high-dimensional feature vector dimension after the spatial feature reconstruction network is the spatial feature reconstruction network, that is, the Y channel information of the light field sub-aperture image array with high angular resolution and high spatial resolution.

[0074] In this embodiment, bilinear upsampling is performed on U and V channel information of a light field sub-aperture array image with low angular resolution and low spatial resolution to obtain U and V channel information of a light field sub-aperture array image with high angular resolution and high spatial resolution.

[0075] In this embodiment, the obtained Y channel information of the light field sub-aperture array image with high angular resolution and high spatial resolution is merged with the U and V channel information of the light field sub-aperture array image with high angular resolution and high spatial resolution, and the merged data is converted into RGB space to obtain an RGB image of the light field sub-aperture array image with high angular resolution and high spatial resolution.

[0076] The light field image spatial angle joint reconstruction method proposed in this embodiment makes full use of the correlation between the light field sub-aperture array image views, and can reconstruct the light field sub-aperture array image with high angular resolution and high spatial resolution at one time. The method described in this embodiment first extracts the Y channel information in the light field sub-aperture array image with low angular resolution and low spatial resolution; secondly, the Y channel information of the light field sub-aperture image array with high angular resolution and low spatial resolution is reconstructed by using the angle feature reconstruction network; then, the Y channel information of the light field sub-aperture image array with high angular resolution and high spatial resolution is obtained through the spatial feature reconstruction network. Subsequently, the U and V channel information of the light field sub-aperture array image with low angular resolution and low spatial resolution is bilinearly upsampled using a bilinear interpolation algorithm to obtain the U and V channel information of the light field sub-aperture array image with high angular resolution and high spatial resolution. Finally, the Y channel information of the final high angular resolution and high spatial resolution light field sub-aperture array image is merged with the U and V channel information of the high angular resolution and high spatial resolution light field sub-aperture array image, and the merged data is converted into RGB space to obtain an RGB image of the high angular resolution and high spatial resolution light field sub-aperture array image. Compared with the traditional light field image angle reconstruction method and the light field image space reconstruction method, this embodiment can deeply explore the correlation between light field sub-aperture array images and improve the reconstruction quality; at the same time, this embodiment can reconstruct the light field sub-aperture array image with high angular resolution and high spatial resolution at one time, thereby improving the reconstruction efficiency.

[0077] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A light field image spatial angle joint reconstruction method, characterized in that: include: Convert the light field sub-aperture image array with low angular resolution and low spatial resolution from RGB space to YUV space and extract the Y channel information; Reconstructing features of Y channel information of a light field sub-aperture image array with low angular resolution and low spatial resolution to obtain Y channel information of a light field sub-aperture image array with high angular resolution and high spatial resolution; Performing bilinear upsampling processing on U channel information and V channel information of a light field sub-aperture array image with low angular resolution and low spatial resolution to obtain U channel information and V channel information of a light field sub-aperture array image with high angular resolution and high spatial resolution; The Y channel information of the light field sub-aperture array with high angular resolution and high spatial resolution is merged with the U channel information and V channel information of the light field sub-aperture array image with high angular resolution and high spatial resolution, and the merged data is converted into RGB space to obtain an RGB image of the light field sub-aperture array image with high angular resolution and high spatial resolution.

2. The method for spatial angle joint reconstruction of light field images according to claim 1, characterized in that: Feature reconstruction of the Y channel information of the light field sub-aperture image array with low angular resolution and low spatial resolution includes: Merging the Y channel information of the light field sub-aperture array with low angular resolution and low spatial resolution to obtain a four-dimensional light field sub-aperture array image with low angular resolution and low spatial resolution; Using an angle feature reconstruction network, the angle feature of the four-dimensional light field sub-aperture array image with low angle resolution and low spatial resolution is reconstructed to obtain Y channel information of the light field sub-aperture array image with high angle resolution and low spatial resolution; The spatial feature reconstruction network is used to reconstruct the Y channel information of the light field sub-aperture array image with high angular resolution and low spatial resolution to obtain the Y channel information of the light field sub-aperture array image with high angular resolution and high spatial resolution.

3. The method for spatial angle joint reconstruction of light field images according to claim 2, characterized in that: The angle feature reconstruction network includes: a feature extraction module, a feature refinement module and a feature enhancement module; The feature extraction module is used to extract the high-dimensional feature vector of the four-dimensional light field sub-aperture array image with low angular resolution and low spatial resolution; The feature refinement module is used to rearrange the dimensions of the high-dimensional feature vector and perform feature refinement processing; The feature enhancement module is used to enhance the high-dimensional feature vector after feature refinement.

4. The method for spatial angle joint reconstruction of light field images according to claim 3, characterized in that: The feature extraction module includes: a first convolution layer, a LeakyReLU excitation layer, a first 3D convolution module, a second convolution layer, a second 3D convolution module, a third convolution layer, a third 3D convolution module, a fourth convolution layer, a fourth 3D convolution module, a fifth convolution layer, a fifth 3D convolution module, a sixth convolution layer, a seventh convolution layer, a first Summation layer, a second Summation layer and a Concatenation layer connected in sequence; wherein the 3D convolution module includes: a first 3D convolution layer, a LeakyReLU excitation layer, a second 3D convolution layer and a Summation layer connected in sequence.

5. The method for spatial angle joint reconstruction of light field images according to claim 3, characterized in that: The feature refinement module includes: four first DE convolution modules connected in sequence; wherein the first DE convolution module includes: DEConv, Conv_1, Conv_2, 2 Summation layers and 1 LeakyReLU excitation layer; The features input into the first DE convolution module first pass through DEConv, and then are summed with the input features through the Summation layer; the summed features pass through Conv_1 and the input features through the Summation layer, and finally pass through Conv_2 to obtain the output of the first DE convolution module.

6. The method for spatial angle joint reconstruction of light field images according to claim 3, characterized in that: The feature enhancement module includes: EN_1, EN_2, LeakyReLU, EN_3, PixelShuffle, EN_4, LeakyReLU and 1 local implicit function; The features of the input feature enhancement module first pass through a local implicit function to obtain implicitly enhanced features, and then pass through EN_1, EN_2, LeakyReLU, EN_3, PixelShuffle, EN_4, and LeakyReLU in sequence to obtain enhanced features.

7. The method for spatial angle joint reconstruction of light field images according to claim 3, characterized in that: The spatial feature reconstruction network includes: a first interaction module, a second interaction module, a third interaction module, a fourth interaction module, a fifth interaction module, a sixth interaction module, UP_1, a PixelShuffle layer, and UP_2 connected in sequence; The features of the input spatial feature reconstruction network are sequentially passed through the first interaction module, the second interaction module, the third interaction module, the fourth interaction module, the fifth interaction module, the sixth interaction module, UP_1, the PixelShuffle layer, and UP_2 to obtain the Y channel information of the light field sub-aperture image array with high spatial resolution and high angular resolution; Each interaction module includes: four second DE convolution modules connected in sequence; wherein the second DE convolution module includes: DEConv, Conv_1, Conv_2, 2 Summation layers and 1 LeakyReLU excitation layer; The features input to the second DE convolution module first pass through DEConv, and then are summed with the input features through the Summation layer. The summed features pass through Conv_1 and the input features through the Summation layer, and finally pass through Conv_2 to obtain the output of the second DE convolution module.

Images