A light field angular domain super-resolution method based on space and angle information fusion

Abstract

The application discloses a light field angle domain super-resolution method based on space and angle information fusion. The steps of the application are as follows: S1: constructing a light field sub-aperture image training dataset and a test dataset; S2: constructing a light field angle super-resolution deep neural network model; S3: designing a loss function for the light field angle super-resolution network model; S4: using the trained light field angle super-resolution deep neural network model to perform a light field angle super-resolution task test on the test dataset. The space-angle information fusion module proposed in the application extracts stronger spatial features in the global range of the light field sub-aperture image through a self-attention mechanism, establishes more specific angle-dependent relationships by using a local implicit function, and generates new space-angle features through fusion, and has stronger information expression capability. The model constructed in the application can reconstruct more accurate results in the light field angle super-resolution task under sparse view angle sampling.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and computer vision, and specifically relates to a light field angle domain super-resolution method based on the fusion of spatial and angular information. Background Technology

[0002] Light field cameras, with their unique acquisition method, can record a complete 4D light field, which includes spatial and angular information of light rays. Light field images have broad application prospects in depth estimation, refocusing, and 3D reconstruction. Since the angular resolution of light field images directly affects their ability to express light field information, improving the angular resolution of light field images has become an important research task in the field of optical imaging.

[0003] Traditional optical field angular super-resolution methods primarily rely on establishing corresponding mathematical models to reconstruct high angular resolution images. These methods depend on the intrinsic parameters of the optical field camera and the depth map of the plaza. In recent years, with advancements in science and technology and improvements in hardware, many deep learning-based optical field super-resolution methods have been proposed. Compared to traditional methods, those reconstructed using deep neural networks offer higher accuracy and richer detail. Some methods extract angular features from sub-aperture images at different viewpoints within the optical field using convolutional neural networks to achieve higher angular resolution reconstruction. Other methods extract continuous angular domain features from epipolar plane images (slices with constant angles and spatial orientations) to generate higher angular resolution optical field images. However, these methods only learn the angular domain information of the optical field, neglecting its spatial domain information. They cannot effectively utilize the rich spatial and angular information of the optical field simultaneously, resulting in poor performance in optical field super-resolution tasks with sparse viewpoint sampling. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a light field angle domain super-resolution method based on the fusion of spatial and angular information.

[0005] To facilitate the extraction of rich spatial and angular domain information from a four-dimensional light field, this invention transforms sub-aperture images into macro-pixel images as input to a deep neural network. Macro-pixel images are obtained by arranging pixels with the same spatial coordinates in sub-aperture images at different angles according to their angular coordinates, and then arranging these macro-pixels according to their spatial coordinates. A light field angular domain super-resolution neural network model is then constructed. This model mainly comprises four spatial-angular information fusion module groups (SA-Groups), each containing three spatial-angular information fusion modules (SA-Blocks). Each SA-Block includes a spatial feature extraction part and an angular feature extraction part. The spatial feature extraction part extracts features from each macro-pixel through 2D convolution and compresses the spatial size of the feature map, then expands the input self-attention layer along the spatial dimension to extract global spatial features. The angular feature extraction part uses local implicit functions to upsample the macro-pixel feature map extracted through 2D convolution, establishing the dependency relationship of angular features. Spatial and angular information is extracted from macro-pixel images using SA-Block to achieve feature fusion. Then, shallow and deep features are superimposed in SA-Group. Finally, features extracted from SA-Group at different stages are fused and upsampled to achieve higher light field angle super-resolution reconstruction.

[0006] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0007] Step S1: Construct the training dataset and test dataset for light field sub-aperture images;

[0008] Step S2: Construct a super-resolution deep neural network model for light field angle;

[0009] Step S3: Design the loss function for the light field angle super-resolution network model;

[0010] Step S4: Using the trained light field angle super-resolution deep neural network model, perform light field angle super-resolution task testing on the test dataset.

[0011] The beneficial effects of this invention are as follows:

[0012] This invention utilizes a proposed spatial-angle information fusion module to extract stronger spatial features in the global scope of the light field sub-aperture image through a self-attention mechanism, establishes more specific angle dependencies using local implicit functions, and then generates new spatial-angle features through fusion, thus having a stronger information expression capability.

[0013] The light field angle super-resolution deep neural network model constructed in this invention can reconstruct more accurate results in the light field angle super-resolution task under sparse view sampling. Attached Figure Description

[0014] Figure 1 A schematic diagram of the training data for the light field image array and the light field sub-aperture image;

[0015] Figure 2 The structure diagram of the light field angle super-resolution deep neural network model;

[0016] Figure 3 This is a schematic diagram of the main steps of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0018] like Figure 3 As shown, a light field angle domain super-resolution method based on the fusion of spatial and angular information includes the following steps:

[0019] Step S1: Construct the training and testing datasets for light field sub-aperture images. The specific method is as follows:

[0020] The training dataset consisted of 20 synthetic scenes from HCIold and HCInew, and all real-world scenes from 100Scenes. The test dataset consisted of 5 synthetic scenes from HCIold, 4 synthetic scenes from HCInew, all real-world scenes from 30scenes, 25 real-world scenes in the Occlusion category and 15 real-world scenes in the Reflective category from the STFlytro dataset.

[0021] For each scene in the light field dataset, it is represented as a set of light field sub-aperture images L(u,v,s,t) using the dual-plane method, where (u,v) are angular coordinates and (s,t) are spatial coordinates.

[0022] Because the spatial resolution of the original light field image is too large, a block-based approach is used to construct the training dataset, such as... Figure 1 As shown, the specific method is as follows: A 7×7 sub-aperture image array of the central region of the light field dataset is used to divide the image into blocks, with a block size of 64×64. The block division step size is half the side length of the block. The four sub-aperture images at the top left, bottom left, top right, and bottom right corners are used as blocks generated from the 2×2 original light field images. These blocks are then recombined according to their angular coordinates to form a low-angle resolution light field image as the input training data. The other blocks at corresponding spatial locations are recombined according to their angular coordinates to form a high-angle resolution light field image as the ground truth label. Figure 1 As shown.

[0023] The specific method for constructing the test dataset is as follows: a 7×7 sub-aperture image array in the central region of the light field dataset is combined according to the angular coordinates to form a high-angle resolution light field image as the real label. The four sub-aperture images in the upper left, lower left, upper right, and lower right corners are combined according to the angular coordinates to form a low-angle resolution light field image as the input test data.

[0024] Step S2: Light field angle super-resolution deep neural network model, such as Figure 2 As shown, the specific method is as follows:

[0025] First, input the sub-aperture image. Convert to macropixel image Then, the initial macro-pixel features are obtained through an initial dilated convolutional layer with a stride of 1, a kernel size of (3,3), and a dilation coefficient of (A,A). C represents the number of output channels of the initial dilated convolutional layer, which is 64.

[0026] The model comprises four cascaded spatial-angle information fusion module groups, each containing three spatial-angle information fusion modules. Initial macro-pixel features. The spatial feature extraction and angular feature extraction components in the spatial-angle information fusion module will be used respectively.

[0027] In the spatial feature extraction part, First, features are obtained by using a standard convolutional layer (PatchEmbed) with a stride of (A,A) and a kernel size of (A,A). d c =C×A×A represents the number of output channels of the convolutional layer. The feature F is reshaped into a two-dimensional sequence containing spatial features. HW represents the number of sequences, d c This represents the feature dimension of each sequence. The spatial position encoding P is obtained by establishing the positional correlation (PositionEncoding) between different spatial features based on spatial coordinates. S As shown in Formula 1, the two-dimensional sequence Token s and spatial location coding P s After addition, Q is obtained through LN layer (layer normalization). S and K S The third input V of the self-attention layer S =Token S .

[0028] Q S =K S =LN(Token)S +P S (1)

[0029] Then, Q S K S and V S The global features of the learning space are fed into the multi-head attention layer (MHSA), as shown in Equation 2:

[0030] T S =MHSA(Q S ,K S V S (2)

[0031] intermediate variable T S Reconstructed into three-dimensional features Transforming spatial features through PixelShuffle

[0032] In the angle feature extraction part, First, features are obtained by using a regular convolutional layer with stride (A,A) and kernel size (A,A). d i =C represents the number of output channels of the convolutional layer. Feature E is obtained through the unfold operation. N = 9 × C, and then the features are obtained through nearest neighbor interpolation upsampling. E P Remodeling into two-dimensional features D = A × A × H × W. F is calculated using matrices. in Generate coordinate matrices for E and E respectively. and The coordinate matrix e is obtained by upsampling through nearest neighbor interpolation to obtain a new coordinate matrix. The local implicit function G consists of a fully connected layer with four output channels of 256 and one output channel of C. An angular feature with established angular dependencies is obtained through G. As shown in Formula 3:

[0033] F A =G(Concat(E) * ,de * (3)

[0034] After the spatial feature extraction and angular feature extraction parts, the spatial feature F is connected along the channel. S and angular features F A Then, spatial-angular information is fused using a regular convolutional layer with a stride of 1 and a kernel size of (1,1) to obtain the initial spatial-angular fused features. Finally, the convolutional layers are divided into two dilated convolutional layers with a stride of 1, a kernel size of (3,3), and a dilation coefficient of (A,A), and then combined with the initial spatial-angular fusion feature F. M Adding them together yields the spatial-angle fusion features.

[0035] Each spatial-angle information fusion module in the spatial-angle information fusion module group receives feature information extracted by all preceding modules through summation. Finally, all intermediate outputs are connected along the channel to complete the feature information fusion from shallow to deep layers.

[0036] Each spatial-angle information fusion module group receives the output of the previous spatial-angle information fusion module group as input, and finally connects all intermediate outputs along the channels. The final features are obtained by passing them through ordinary convolutional layers with a stride of 1 and a kernel size of (1,1).

[0037] In the final upsampling stage, the final feature F * First, downsampling features are obtained by downsampling using a regular convolutional layer with stride (A,A) and kernel size (A,A). Downsampling features First, convolutional features are obtained by setting the target's high-angle resolution to B×B, stride to 1, and kernel size to (1,1) using a regular convolutional layer. J = C × B × B, and intermediate features are obtained through PixelShuffle. Then the intermediate feature F U A high-angular-resolution macro-pixel image of the target is obtained by using a dilated convolutional layer with a stride of 1, a kernel size of (3,3), a dilation coefficient of (A,A), and an output channel equal to the target channel value. Finally, the macropixel image Convert to sub-aperture image

[0038] Step S3: Design the loss function for the light field angle super-resolution network;

[0039] Output high-angle resolution sub-aperture image and true high-angle resolution sub-aperture images The formula for calculating the loss function Loss is shown in Formula 4:

[0040]

[0041] Step S4: Using the trained light field angle super-resolution deep neural network model, perform light field angle super-resolution task testing on the test dataset, as follows:

[0042] The low-angle resolution light field sub-aperture images for each scene in the test dataset are divided into blocks according to their spatial location. The block division rule is as follows: the block division step size is half the block side length, and the top, bottom, left, and right boundaries of each sub-aperture image are zero-padded to ensure that the number of blocks is a square integer. The blocks obtained from each scene division are as follows: H = W = 64, where s and t represent the relative position of the block within the sub-aperture image space. Low-angular-resolution blocks are then used. The data is sequentially input into the trained light field angle super-resolution deep neural network model to obtain high-angle resolution blocks. Once all blocks have been processed, based on the spatial relative positions represented by s and t in each high-angle resolution block, only the central region of that block is retained as the prediction result of the network model. The side length of the central region is half the side length of the block. Finally, a complete high-angle resolution light field sub-aperture image is synthesized. Then, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were calculated using high-angle resolution light field sub-aperture images as test metrics.

[0043] Table 1 Comparison with other methods in 2×2 to 7×7 tasks (PSNR / SSIM)

[0044] <![CDATA[Kalantari [1] ]]> 41.40 / 0.982 37.25 / 0.972 38.09 / 0.953 32.85 / 0.909 38.58 / 0.944 <![CDATA[Wuetal [2] ]]> 38.58 / 0.944 32.72 / 0.924 34.76 / 0.930 26.64 / 0.744 31.43 / 0.850 <![CDATA[Jinetal [3] ]]> 42.54 / 0.986 38.53 / 0.979 38.46 / 0.959 34.60 / 0.937 40.84 / 0.960 <![CDATA[LF-EASR [4] ]]> 43.44 / 0.989 39.80 / 0.985 39.35 / 0.963 35.86 / 0.956 41.54 / 0.960 Ours 44.16 / 0.995 40.11 / 0.992 39.36 / 0.979 35.93 / 0.975 41.60 / 0.975

[0045] Furthermore, the present invention also provides a light field angle domain super-resolution system based on the fusion of spatial and angular information, comprising the following modules:

[0046] Data acquisition module: Constructs training and testing datasets for light field sub-aperture images;

[0047] Model building module: Constructs a super-resolution deep neural network model of light field angle;

[0048] Loss module: Designs the loss function for the light field angle super-resolution network model;

[0049] Test module: Using the trained light field angle super-resolution deep neural network model, perform light field angle super-resolution task tests on the test dataset.

[0050] Furthermore, the model building module first transforms the input sub-aperture image into a macro-pixel image; secondly, the model comprises four spatial-angle information fusion module groups, cascaded sequentially, with each spatial-angle information fusion module group containing three spatial-angle information fusion modules. The initial macro-pixel features are extracted through the spatial feature extraction and angle feature extraction parts of the spatial-angle information fusion modules, respectively.

[0051] Loss module: The loss function Loss between the output high-angle resolution sub-aperture image and the true high-angle resolution sub-aperture image.

[0052] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0053] The parts of this invention not described in detail are well-known to those skilled in the art.

[0054] The references for other methods are as follows:

[0055] [1]Kalantari NK,Wang TC,Ramamoorthi R.Learning-based view synthesis for light field cameras[J].ACM Transactions on Graphics(TOG),2016,35(6):1-10.

[0056] [2]Wu G, Liu Y, Dai Q, et al. Learning sheared EPI structure for lightfield reconstruction[J]. IEEE Transactions on Image Processing, 2019, 28(7): 3261-3273.

[0057] [3]Jing J,Hou J,Yuan H,et al.Learning light field angular super-resolution via a geometry-aware network[C] / / Proceedings of the AAAIconference on artificial intelligence.2020,34(07):11141-11148.[4]Liu G,Yue H,Wu J,et al.Efficient light field angular super-resolution with sub-aperturefeature learning and macro-pixel upsampling[J].IEEE Transactions onMultimedia,2022。

Claims

1. A light field angle domain super-resolution method based on the fusion of spatial and angular information, characterized in that... Includes the following steps: Step S1: Construct the training dataset and test dataset for light field sub-aperture images; Step S2: Construct a super-resolution deep neural network model for light field angle; Step S3: Design the loss function for the light field angle super-resolution network model; Step S4: Using the trained light field angle super-resolution deep neural network model, perform light field angle super-resolution task testing on the test dataset; Step S2: Light field angle super-resolution deep neural network model, the specific method is as follows: First, input the sub-aperture image. Convert to macropixel image Then, the initial macro-pixel features are obtained through an initial dilated convolutional layer with a stride of 1, a kernel size of (3,3), and a dilation coefficient of (A,A). , This indicates the initial number of output channels for the dilated convolutional layer, which is 64. The model comprises four spatial-angle information fusion module groups, cascaded sequentially, with each spatial-angle information fusion module group containing three spatial-angle information fusion modules; initial macro-pixel features The spatial feature extraction and angular feature extraction components in the spatial-angle information fusion module will be used respectively; In the spatial feature extraction section, First, features are obtained by using a regular convolutional layer with stride (A,A) and kernel size (A,A). , Indicates the number of output channels of the convolutional layer; [the text abruptly ends here, likely due to an in Reconstructed into a two-dimensional sequence containing spatial features , Represents the number of sequences. This represents the feature dimension of each sequence; spatial location encoding is obtained by establishing the positional correlation between different spatial features based on spatial location coordinates. ; As shown in Formula 1, the two-dimensional sequence and spatial location coding After addition, the result is obtained through the LN layer. and The third input to the self-attention layer ; ； Then, , and The global features of the learning space are fed into the multi-head attention layer, as shown in Equation 2: ； intermediate variables Reconstructed into three-dimensional features Transformed into spatial features through PixelShuffle .

2. The optical field angle domain super-resolution method based on spatial and angular information fusion according to claim 1, characterized in that... Step S1 involves constructing the training and testing datasets for the light field sub-aperture images. The specific method is as follows: The training dataset was sourced from 20 synthetic scenes from HCIold and HCInew, and all real scenes from 100Scenes; the test dataset was sourced from 5 synthetic scenes from HCIold, 4 synthetic scenes from HCInew, all real scenes from 30scenes, 25 real scenes in the Occlusion category and 15 real scenes in the Reflective category of the STFlytro dataset.

3. The optical field angular domain super-resolution method based on spatial and angular information fusion according to claim 1, characterized in that... For each scene in the light field dataset, the dual-plane method is used to represent it as a set of light field sub-aperture images. ,in For angular coordinates, Spatial coordinates; Therefore, a block-based approach is used to construct the training dataset. Specifically, the central region of the light field dataset is used as the data source. The sub-aperture image array is divided into blocks, with a block size of [missing value]. The partitioning step size is half the side length of the partition, where the four sub-aperture images at the top left, bottom left, top right, and bottom right are used as... The blocks generated from the original light field image are recombined into a low-angle resolution light field image based on the angular coordinates as the input training data, while the blocks with other corresponding spatial positions are recombined into a high-angle resolution light field image based on the angular coordinates as the true labels. The specific method for constructing the test dataset is as follows: using the central region of the light field dataset... The sub-aperture image array is combined according to angular coordinates to form a high-angular-resolution light field image as the ground truth label, wherein the four sub-aperture images at the top left, bottom left, top right, and bottom right are used as... The light field images are combined according to the angular coordinates to form a low-angular-resolution light field image, which is used as the input test data.

4. The optical field angular domain super-resolution method based on spatial and angular information fusion according to claim 2, characterized in that... In the angle feature extraction part, First, features are obtained by using a regular convolutional layer with stride (A,A) and kernel size (A,A). , Indicates the number of output channels of the convolutional layer; features Features are obtained through the unfold operation. , Then, features are obtained through nearest neighbor interpolation upsampling. ,Will Remodeling into two-dimensional features ; through matrix calculation and Generate coordinate matrices respectively and coordinate matrix The new coordinate matrix is ​​obtained through nearest neighbor interpolation upsampling. Local implicit functions It consists of four output channels with a total of 256 and one output channel with a total of 1000. Composed of fully connected layers, through Angular features with established angular dependencies were obtained. As shown in Formula 3: 。 5. The optical field angular domain super-resolution method based on spatial and angular information fusion according to claim 3, characterized in that... After the spatial feature extraction and angular feature extraction parts, the spatial features are connected along the channel. and angular features Then, spatial-angular information is fused using a regular convolutional layer with a stride of 1 and a kernel size of (1,1) to obtain the initial spatial-angular fused features. Finally, the initial spatial-angular fusion features are passed through two dilated convolutional layers with a stride of 1, a kernel size of (3,3), and a dilation coefficient of (A,A). Adding them together yields the spatial-angle fusion features. Each spatial-angle information fusion module in the spatial-angle information fusion module group receives the feature information extracted by all the previous modules by adding them together; finally, all intermediate outputs are connected along the channel to complete the feature information fusion from shallow to deep layers. Each spatial-angle information fusion module group receives the output of the previous spatial-angle information fusion module group as input, and finally connects all intermediate outputs along the channels. The final features are obtained by passing them through ordinary convolutional layers with a stride of 1 and a kernel size of (1,1). In the final upsampling stage, the final features First, downsampling features are obtained by downsampling using a regular convolutional layer with stride (A,A) and kernel size (A,A). Downsampling features By setting the target high-angle resolution as A regular convolutional layer with a stride of 1 and a kernel size of (1,1) first obtains the convolutional features. Intermediate features are obtained through PixelShuffle. Then intermediate features A high-angular-resolution macro-pixel image of the target is obtained by using a dilated convolutional layer with a stride of 1, a kernel size of (3,3), a dilation coefficient of (A,A), and an output channel equal to the target channel value. Finally, the macro-pixel image Convert to sub-aperture image .

6. The optical field angular domain super-resolution method based on spatial and angular information fusion according to claim 5, characterized in that... Step S3: Design the loss function for the light field angle super-resolution network, and implement it as follows: Output high-angle resolution sub-aperture image and true high-angle resolution sub-aperture images loss function between The calculation formula is shown in Formula 4: 。 7. A super-resolution system in the optical field angle domain based on the fusion of spatial and angular information, characterized in that, The system is used to implement the method as described in claim 1, and includes the following modules: Data acquisition module: Constructs training and testing datasets for light field sub-aperture images; Model building module: Constructs a super-resolution deep neural network model of light field angle; Loss module: Designs the loss function for the light field angle super-resolution network model; Test module: Using the trained light field angle super-resolution deep neural network model, perform light field angle super-resolution task tests on the test dataset.

8. A super-resolution system in the optical field angle domain based on the fusion of spatial and angular information according to claim 7, characterized in that: Model building module: First, the input sub-aperture image is transformed into a macro-pixel image; second, the model contains a total of four spatial-angle information fusion module groups, which are cascaded in sequence. Each spatial-angle information fusion module group contains three spatial-angle information fusion modules. The initial macro-pixel features will be extracted through the spatial feature extraction part and the angle feature extraction part in the spatial-angle information fusion module, respectively. Loss module: The loss function between the output high-angle resolution sub-aperture image and the true high-angle resolution sub-aperture image. .

Images