A method for generating super-resolution light field images based on multi-view information interaction

Abstract

This invention discloses a light field super-resolution network generation method based on multi-view information interaction, belonging to the field of image super-resolution reconstruction technology, aiming to solve the problem of insufficient spatial resolution in light field images. This invention employs a generative adversarial network (GAN) as the overall network structure to improve network performance. Considering the specific characteristics of light field data, decomposition kernel convolution is used to extract spatial angular information from the light field image, and this is combined with a channel attention mechanism to form a channel attention residual module based on the decomposition kernel. The generator of the proposed network uses multiple decomposition kernel channel attention residual modules connected in series, and a dense input residual structure is introduced in this part to achieve feature extraction. The discriminator also uses decomposition kernel convolution to extract features and outputs the final discrimination result. This network can extract richer information from light field data to achieve light field super-resolution reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of computational imaging technology, and in particular relates to a method for generating super-resolution light field images based on multi-view information interaction via a network. Background Technology

[0002] With the development of technology, the field of camera imaging is also rapidly evolving. Traditional cameras, when photographing objects, only record the 2D projection of light, thus losing a significant amount of spatial angular information about the light field. To obtain this missing information, it is often necessary to take multiple shots of the scene from different locations to obtain multi-view images. Compared to traditional cameras, light field cameras can obtain multi-view images of a scene with just one shot, simultaneously acquiring both spatial and angular information of the light field. After processing, they can provide depth information, refocused viewpoint maps, and other data, resulting in higher imaging efficiency than traditional cameras. However, the cost of acquiring angular information with light field cameras is the sacrifice of spatial resolution in a single viewpoint image. Low spatial resolution significantly impacts subsequent data processing. Therefore, super-resolution technology for light field images is of great importance.

[0003] In recent years, due to the development of Convolutional Neural Networks (CNNs) and the emergence of light field datasets, deep learning-based methods have achieved good performance in light field reconstruction. In 2016, Yoon et al. first applied CNNs to light field super-resolution reconstruction, using two cascaded CNNs to perform spatial and angular super-resolution reconstruction of light field images. In 2017, Farrugia et al. proposed a dictionary-based learning method to learn the mapping between low-resolution and high-resolution light field images. In the same year, Gaochang Wu et al. from Tsinghua University studied light field super-resolution reconstruction techniques based on epipolar plane images (EPI). In 2018, Wang et al. modified the neural network structure, designing a bidirectional recursive CNN, and then used the CNN to perform super-resolution reconstruction on horizontal and vertical image stacks separately, finally unifying the stacks through superposition generalization to obtain a complete view image. Zhang et al. used a multi-branch residual network to achieve spatial super-resolution reconstruction of the light field. The inputs to different branches were sub-aperture images stacked in different directions, which were used to learn the correlation between the sub-aperture images in different directions. Finally, the extracted features were fused to reconstruct the light field. In 2020, Chen et al. applied the structure of Generative Adversarial Network (GAN) to the super-resolution reconstruction of the light field and proposed the EPI loss function to reduce the gap between the reconstruction result and the real light field.

[0004] On the other hand, numerous improvements to convolution methods have been proposed in recent years. In 2019, Meng et al. proposed a convolution method for four-dimensional light field data, namely 4D convolution, which extracts spatial angular information of the light field simultaneously through high-dimensional convolution. In the same year, Yeung et al. addressed the low computational efficiency of 4D convolution by proposing Spatial-Angular Separable (SAS) convolution, which significantly improved the computational speed of the network while maintaining performance comparable to 4D convolution. In 2022, Hu et al. further improved the dimensionality of light field data based on separable convolution by proposing the concept of a decomposition kernel. This convolution method can not only extract information from the angular and spatial subspaces of the light field like SAS convolution, but also further extract information from the four EPI subspaces of the light field. Besides separable convolution methods, in 2020, Wang et al. proposed the spatial feature extractor (SFE) and angular feature extractor (AFE), which extract spatial and angular information from light field images by changing the shape of 2D convolutions. In 2022, the authors further proposed horizontal and vertical EPI feature extractors based on SFE and AFE, further improving the network's performance. It's worth noting that the only improvement in convolution direction for light field GANs so far is LightGAN proposed by Meng et al. in 2020. This method replaces 2D convolutions with 4D convolutions in the GAN network to extract high-dimensional information from light field data, achieving light field reconstruction.

[0005] In recent years, deep learning has made great progress in the field of light field image super-resolution, but there is still insufficient utilization of light field data information, and many methods do not pay attention to the information of light field data in the channel dimension during the super-resolution process. Summary of the Invention

[0006] This invention addresses the problems existing in current deep learning methods by providing a light field image super-resolution network generation method based on multi-view information interaction. When performing light field image super-resolution, the network of this invention can more fully extract complementary information between light field images from different viewpoints to apply to the super-resolution reconstruction of the light field. Furthermore, it introduces a channel attention mechanism and an adversarial generative network structure.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A method for generating super-resolution light field images based on multi-view information interaction is proposed. The inputs to both the generator and discriminator are the original 4D data of the light field, i.e., angular resolution U×V and spatial resolution W×H. The generator first extracts features through 3×3 spatial convolutions and eight channel attention residual modules (CARs) based on decomposition kernels, introducing input image concatenation and dense residual connections in this part. Then, two angular convolutions are used for feature fusion, and finally, sub-pixel convolutions are used to upsample the spatial resolution. The super-resolution light field image output by the generator and the original high-resolution light field image are then fed into the discriminator. The discriminator first extracts image features through eight layers of decomposition kernel convolutions, and finally outputs the discrimination result after an average pooling layer and two 1×1 convolutions. The specific processes of each step are as follows:

[0009] The specific design of the decomposition kernel convolution mentioned in the generator is as follows: the input light field image has a five-dimensional data dimension (u, v, w, h, c), where c represents the number of channels. After dimensionality reduction, the light field image is obtained in various low-dimensional subspaces. Light field data with dimensions (u×v, w, h, c) is considered a sub-aperture image, and u×v is considered a batch of viewpoint images, also called a spatial subspace. Similarly, light field data with dimensions (w×h, u, v, c) is considered an angle subspace, and light field data with dimensions (u×h, w, v, c), (w×v, u, h, c), (v×h, u, w, c), and (w×u, h, v, c) are considered EPI subspaces, for a total of six subspaces. The operation of the decomposition kernel is to reduce the light field data to each subspace sequentially, and then perform a 2D convolution operation on the 2-dimensional and 3-dimensional data, which can be expressed by the following formula:

[0010] DKConv=k h，v (k u，w (k u，h (k w，v (k u，v (k w，h (L))))))

[0011] Where L represents the original five-dimensional data of the light field, k w.h This represents convolution over a spatial subspace; this convolution is also called spatial convolution. u，v This represents convolution over an angular subspace, also known as angular convolution, where k w，v k u，h k u，w k h，v This indicates convolution over the EPI subspace, where DKConv represents the decomposition kernel. The specific convolution operation can be expressed by the following formula:

[0012]

[0013] in, This represents a 2D convolution operation, where W represents the weights of the corresponding 2D convolution. The corresponding subspace representing the deformation of the original five-dimensional light field data L, and f(·) represents the ReLU activation function operation; This represents the operation of deforming the light field data and performing convolution in the dimension (d1, d2).

[0014] Furthermore, the structure of the Channel Attention Residual Module (CAR) based on the decomposition kernel is as follows: First, a 1×1 decomposition kernel convolution is placed before the module as a bottleneck layer (BNL) to reduce the number of input channels of the module to a preset size. After passing through the feature extraction layer of the 3×3 decomposition kernel convolution, the module is fed into the channel attention module to calculate the channel attention map. Then, the obtained channel attention is weighted and multiplied and added to the input of the channel attention module. Finally, it is added to the input of the entire module to obtain the final output of the module.

[0015] The channel attention module mentioned above operates as follows: The input light field data is passed through a global pooling layer. Since channel attention is being calculated, this pooling layer will affect all dimensions of the data except for the channels, namely the spatial dimension, angular dimension, and so on. Then, the number of dimensions is reduced to only the channel dimension. Finally, a downsampling 1×1 convolution and an upsampling 1×1 convolution are applied to reduce the channel dimension of the data to C. r Then, we return to C, using this step to predict the weights of each channel in the feature map. A PReLU activation function is set between the two convolutions, and finally, after passing through the sigmoid activation function, the data dimension is expanded to match the input data, thus obtaining the channel attention map of the input feature map.

[0016] The overall feature extraction process is as follows: After the network generator receives the light field data, it first passes through a 3×3 spatial convolution to expand the number of channels to a preset size. Then, it passes through the feature extraction part, which consists of 8 decomposition kernel channel attention modules connected in series. The input of each module is composed of the generator input and the output of all the modules before it, connected in the channel dimension. That is, it is a structure of input dense residual connections composed of original image connections and dense connections. This can make full use of the feature map information of each layer of the network and also reduce the gradient vanishing phenomenon of the network.

[0017] The final steps for generating the light field super-resolution image in the network are as follows: The feature map obtained by the feature extraction module is first processed by a 3×3 angular convolution to fuse and reduce the feature map of a large number of channels due to dense residual connections to a preset channel size, so as to fuse the information between the various viewpoints. Then, it is processed by a 3×3 spatial convolution to expand the number of channels to a square of the upsampling factor of the network input image channels, i.e., a.2 c is used to prepare for subpixel convolution operations. Finally, after subpixel convolution, pixels at the same position in the feature maps of each channel are arranged and fused in an orderly manner to achieve an upsampling of the spatial resolution of the light field image to a times that of the input light field image.

[0018] Besides the generator, the discriminator, another part of the network structure, follows these steps: The super-resolution light field image output by the generator and the original high-resolution light field image are then fed into the discriminator. First, image features are extracted through eight layers of decomposition kernel convolutions. All decomposition kernels are 3×3 in size. Except for the first decomposition kernel, the output channels of odd-numbered decomposition kernel convolutional layers are twice the number of input channels, with a stride of 1. The output channels of even-numbered decomposition kernel convolutional layers are the same as the number of input channels, but with a stride of 2, ensuring that the feature map size is halved each time the number of channels doubles. All decomposition kernels except the first are followed by a batch normalization (BN) layer. Finally, after an average pooling layer and two 1×1 convolutions, the discriminator outputs the discrimination result after processing by the sigmoid activation function. The closer the discrimination result is to 1, the higher the probability that the discriminator considers the input image to be a high-resolution image; the closer the discrimination result is to 0, the higher the probability that the discriminator considers the input image to be a generated super-resolution image. This discrimination result is fed back to the generator, guiding its improvement.

[0019] The advantages and positive effects of this invention are as follows: In the feature extraction part of the generator, this invention employs decomposed convolution kernels to adapt to the 4D information of the light field. Compared to ordinary 2D convolution, this can better extract the spatial and angular information of the light field. Simultaneously, a channel attention mechanism is introduced, incorporating channel-level information into the feature extraction process, achieving more comprehensive feature extraction and fusion of the light field data. Furthermore, the presence of dense residual connections in the input makes the transfer of features and gradients in the generator more efficient, making the generator easier to train. The discriminator borrows the discriminator structure from SRGAN, ensuring a balance between the discriminator and generator to obtain better generation results. Compared with other light field super-resolution algorithms, it achieves better performance. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the generator network structure of the network of the present invention.

[0021] Figure 2 This is a schematic diagram of the bottleneck layer in the generator.

[0022] Figure 3 This is a schematic diagram of the channel attention residual module based on the decomposition kernel.

[0023] Figure 4 This is a schematic diagram illustrating the process of decomposing kernel convolution.

[0024] Figure 5 This is a schematic diagram of the discriminator structure of the network of the present invention. Detailed Implementation

[0025] To make the features and technical solutions of the present invention clearer and easier to understand, the specific embodiments of the present invention will be described in more detail below.

[0026] Network generators, such as Figure 1 As shown, the network's input is a low-resolution light field image (LR) obtained by downsampling a high-resolution light field image (HR), with an angular resolution of U×V and a spatial resolution of W×H. The network consists of two parts: feature extraction and image generation. The feature extraction part first performs shallow feature extraction through a single convolutional layer, followed by deep feature extraction using eight channel attention-based residual layers (CAR) based on decomposition kernels. To reduce the gradient vanishing problem, dense residual connections are introduced in the feature extraction part. Therefore, this invention adds a bottleneck layer (BNL) before the CAR module to ensure that the number of input feature channels of the CAR module is consistent with the preset. Both the BNL and CAR modules use decomposition kernel convolution (DKConv) to adapt to the 4D data of the light field. Finally, the generator outputs a super-resolution image (SR). The following is a detailed description of each part of the entire network.

[0027] Feature extraction based on CAR module

[0028] After receiving the light field data, the network generator first expands the number of channels of the data to a preset size through a 3×3 convolution (conv). Then, it sequentially passes through the feature extraction part, which consists of 8 CAR modules connected in series. The input of each module is composed of the generator input and the output of all the modules before it, connected in the channel dimension. This structure of dense residual connections, which consists of connections of the original image and dense connections, can make full use of the feature map information of each layer of the network and can also reduce the gradient vanishing phenomenon of the network.

[0029] Bottleneck Layer (BNL)

[0030] Because the feature extraction part uses a dense residual structure, we added a bottleneck layer (BNL) before the CAR module, such as... Figure 2 As shown, this layer consists of a 1×1 decomposed kernel convolution to ensure that the number of channels in the input data of each module remains consistent.

[0031] Channel Attention Residual Module (CAR) Based on Decomposition Kernel

[0032] While decomposition kernel convolution fully extracts information from the spatial and angular dimensions of the original light field data, it neglects the channel dimension information. Information between feature maps of different dimensions also plays a positive role in the super-resolution reconstruction of the light field. Therefore, this invention connects a channel attention module after the decomposition kernel convolution to obtain the weights of different channels, thereby capturing information between different feature maps. A residual structure is added to this module to ensure that the gradient of the network does not vanish, thereby reducing the training difficulty of the network and improving the network performance.

[0033] The specific process of a single CAR module: as follows Figure 3 As shown, feature extraction is first performed using a 3×3 decomposed kernel convolution, and then fed into the channel attention module. This module passes the input feature map through a global pooling layer. Since channel attention is being calculated, this pooling layer will affect all dimensions of the feature map except for the channels, i.e., the spatial dimension, angular dimension, and four other dimensions. Afterwards, a reshape is performed to reduce the number of dimensions, retaining only the channel dimension. Then, a 1×1 convolution (conv) with downsampling and upsampling is performed to reduce the channel dimension of the data to C. r Then, the process is restored to C, which is used to predict the weights of each channel in the feature map. A PReLU activation function is set between the two convolutions, and finally, after passing through a sigmoid activation function, the data dimensions are expanded to match the input data through reshaping, thus obtaining the channel attention map of the input feature map. The obtained channel attention maps are then weighted and multiplied and added to the input of the channel attention module. Finally, this is added to and concatenated with the input of the entire module to obtain the final output of the module.

[0034] Decomposed kernel convolution (DKConv)

[0035] When reconstructing a light field, the complementary information between images from different perspectives plays a very positive role in the reconstruction result. Therefore, in order to extract the complementary information from multiple perspective images, this invention uses decomposition kernel convolution to adapt to the high-dimensional data of the light field in order to achieve information extraction from the light field.

[0036] Before introducing the decomposition kernel, we first analyze the principle of dimensionality reduction for light field data. The original light field data has five dimensions: angular dimension (u, v), spatial dimension (w, h), and channel c. When reducing a 4D light field image to a 2D image, the product of any two of the four dimensions (angular and spatial) is used as the array number of the 2D image. The remaining three dimensions are represented as a single 2D image, thus completing the dimensionality reduction operation. Here, light field data with dimensions (u×v, w, h, c) is considered a sub-aperture image array, where u×v is considered a viewpoint image batch, also known as a spatial subspace. Similarly, light field data with dimensions (w×h, u, v, c) is the angular subspace, and light field data with dimensions (u×h, w, v, c), (w×v, u, h, c), (v×h, u, w, c), and (w×u, h, v, c) are the EPI subspaces, for a total of six subspaces.

[0037] The specific process of single decomposition kernel convolution is as follows: Figure 4 As shown, the input light field image has five dimensions (u, v, w, h, c). First, this data is reshaped to (u×v, w, h, c), and then a 3×3 2D convolution (conv) is performed on the (w, h) dimensions, followed by a ReLU activation function. The next five steps sequentially transform the light field data to the other five data formats, then perform 3×3 convolution and ReLU activation operations on them, finally restoring it to the original five-dimensional light field data format. This can be represented by the following formula:

[0038] DKConv=k h，v (k u，w (k u，h (k w，v (k u， v(k w，h (L))))))

[0039] Where L represents the original five-dimensional data of the light field, k w.h This represents convolution over a spatial subspace; this convolution is also called spatial convolution. u，v This represents convolution over an angular subspace, also known as angular convolution, where k w，v k u，h k u，w k h，v This indicates convolution over the EPI subspace, where DKConv represents the decomposition kernel. The specific convolution operation can be expressed by the following formula:

[0040]

[0041] in, This represents a 2D convolution operation, where W represents the weights of the corresponding 2D convolution. The corresponding subspace representing the deformation of the original five-dimensional light field data L, and f(·) represents the ReLU activation function operation; This represents the operation of deforming the light field data and performing convolution in the dimension (d1, d2).

[0042] Image generation section

[0043] The feature map obtained by the feature extraction module is first processed by a 3×3 convolution (conv) to perform feature fusion and reduction on the multi-channel feature map obtained from dense residual connections, reducing the number of feature channels to a preset size to fuse information between different viewpoints. Then, it is processed by another 3×3 convolution (conv) to expand the number of channels to a squared multiple of the upsampling factor of the network input image channels, i.e., a. 2 c is used to prepare for subpixel convolution operations. Finally, after passing through the subpixel convolution layer (Pixel Shuffler), the pixels at the same position in the feature maps of each channel are arranged and fused in an orderly manner to achieve an upsampling of the spatial resolution of the light field image to a times that of the input light field image.

[0044] Discriminator Network

[0045] The discriminator plays a crucial role in generative adversarial networks (GANs). The discriminator's result indicates the direction of the generator's optimization; therefore, the discriminator and generator need to maintain a balance. If either becomes too powerful, the generator will lose its optimization objective. The discriminator of this invention is as follows: Figure 5 As shown, the SR and HR images generated by the generator are used as input. The main body of the network also uses decomposition kernel convolution to extract information from the light field image. The output of the discriminator is fed back to the generator to guide its optimization. The specific process of the network is as follows:

[0046] The super-resolution light field image output by the generator and the original high-resolution light field image are fed into the discriminator. First, image features are extracted through eight layers of decomposition kernel convolutions. Except for the first decomposition kernel, all decomposition kernels are followed by a batch normalization (BN) layer. All decomposition kernel convolutions are 3×3 in size. The number of feature channels in this part of the kernel convolution gradually doubles from 24 to 192. Each time the number of feature channels doubles, the stride of the decomposition kernel convolution is set to 2, reducing the size of the feature map to half its original size. The data output from the final decomposition kernel convolution layer is processed by an average pooling layer (AvgPool) and two 1×1 convolutions (conv), then processed by a sigmoid activation function to output the discrimination result. The closer the discrimination result value is to 1, the higher the probability that the discriminator considers the input image to be a high-resolution image; the closer the discrimination result value is to 0, the higher the probability that the discriminator considers the input image to be the generated super-resolution image. This discrimination result is fed back to the generator, guiding its improvement.

[0047] This invention was trained and tested on the STFlytro public dataset. First, a 5×5 viewpoint image of the center of the original light field image was selected. Then, the RGB image of the original light field was converted into a YCbCr image. A low-resolution image was generated by bicubic interpolation downsampling. The Y channel image was then fed into the network for learning to generate a high-resolution image. The remaining CbCr channels were directly upsampled using the bicubic interpolation method. Finally, the resulting Y channel image and CbCr image were merged to obtain the final high-resolution light field image. During training, 48×48 blocks were cropped and randomly rotated to serve as the training dataset. The network was built using a PyTorch architecture and trained using a loss function that combines perceptual, adversarial, and MSE loss functions. The Adam method was used for optimization, with a batch size of 2 and an initial learning rate of 10 for both the generator and discriminator. -4 The learning rate is halved every 100 cycles, and training ends after 700 cycles.

[0048] This invention proposes a light field image super-resolution network based on multi-view information interaction to address the problem of insufficient spatial resolution in light field images. The network first employs a generative adversarial network to enhance its learning ability. Then, considering the unique characteristics of 4D light fields, decomposed kernel convolution is used to extract spatial angular information of the light field, and a channel attention mechanism is introduced to further extract information from the light field data. The effectiveness of this invention is verified on publicly available datasets. Furthermore, to address the issue of low SSIM values ​​in the network results, the loss function structure will be further optimized in future projects to achieve better results while maintaining network performance.

[0049] The embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for generating a light field super-resolution network based on multi-view information interaction, characterized in that: The inputs to both the network generator and discriminator are the original 4D data of the light field, i.e., angular resolution U×V and spatial resolution W×H. The network generator first extracts features through 3×3 spatial convolution and 8 channel attention residual modules based on decomposition kernels, and introduces input image connections and dense residual connections. Then, it uses 2 angular convolutions to achieve feature fusion, and finally uses subpixel convolution to complete spatial resolution upsampling. The super-resolution light field image output by the generator and the original high-resolution light field image are then fed into the discriminator. First, image features are extracted through 8 layers of decomposition kernel convolution, and finally, the discriminant results are output after one average pooling layer and two 1×1 convolutions. The specific design of the decomposition kernel convolution is as follows: the input light field image has a five-dimensional data dimension (u,v,w,h,c), where c represents the number of channels; after dimensionality reduction, the light field image is obtained in various low-dimensional subspaces. Light field data with dimensions (u×v,w,h,c) is considered a sub-aperture image, and u×v is considered a batch of viewpoint images, also called a spatial subspace; similarly, light field data with dimensions (w×h,u,v,c) is the angle subspace, and light field data with dimensions (u×h,w,v,c), (w×v,u,h,c), (v×h,u,w,c), and (w×u,h,v,c) are the EPI subspaces, for a total of six subspaces. The operation of the decomposition kernel is to reduce the light field data to each subspace sequentially, and then perform 2D convolution operations on the 2- and 3-dimensional data, which can be expressed by the following formula: DKConv=k h,v (k u,w (k u,h (k w,v (k u,v (k w,h (L)))))) Where L represents the original five-dimensional data of the light field, k w,h This represents convolution over a spatial subspace; this convolution is also called spatial convolution. u,v This represents convolution over an angular subspace, also known as angular convolution, where k w,v k u,h k u,w k h,v This indicates convolution over the EPI subspace, and DKConv represents the decomposition kernel; The specific convolution operation is represented by the following formula: in, This represents a 2D convolution operation, where W represents the weights of the corresponding 2D convolution. The corresponding subspace representing the deformation of the original five-dimensional light field data L, and f(·) represents the ReLU activation function operation; This represents the operation of deforming the light field data and performing convolution in the dimension (d1, d2); The structure of the channel attention residual module based on the decomposition kernel is as follows: First, a 1×1 decomposition kernel convolution is placed before the channel attention residual module as a bottleneck layer to reduce the number of input channels of the channel attention residual module to a preset size. After passing through the feature extraction layer of the 3×3 decomposition kernel convolution, the input is fed into the channel attention module to calculate the channel attention map. Then, the obtained channel attention is weighted and multiplied and added to the input of the channel attention module. Finally, it is added to the input of the entire channel attention residual module to obtain the final output of the channel attention residual module.

2. The method for generating a light field super-resolution network based on multi-view information interaction according to claim 1, characterized in that: The channel attention module works as follows: The input light field data is passed through a global pooling layer. Since channel attention is being calculated, this pooling layer affects all dimensions of the data except for the channels, i.e., the spatial and angular dimensions. Then, the number of dimensions is reduced to only the channel dimension. Finally, the data is passed through a downsampling 1×1 convolution and an upsampling 1×1 convolution to reduce the channel dimension of the data to C. r Then, we return to C, using this step to predict the weights of each channel in the feature map. A PReLU activation function is set between the two convolutions, and finally, after passing through the sigmoid activation function, the data dimension is expanded to match the input data, thus obtaining the channel attention map of the input feature map.

3. The method for generating a light field super-resolution network based on multi-view information interaction according to claim 1, characterized in that: The feature extraction process is as follows: After the network generator receives the light field data, it first expands the number of channels of the data to a preset size through a 3×3 spatial convolution. Then, it passes through the feature extraction part, which consists of 8 bottleneck layers and channel attention residual modules connected in series. The input of each channel attention residual module is composed of the generator input and the output before the channel attention residual module, which are connected in the channel dimension. That is, the structure of input dense residual connection is composed of the original image connection and dense connection.

4. The method for generating a light field super-resolution network based on multi-view information interaction as described in claim 1, characterized in that: The steps for generating super-resolution light field images are as follows: First, the obtained feature maps are processed by a 3×3 angular convolution to fuse and reduce the feature maps of a large number of channels with dense residual connections to a preset channel size, thereby fusing information between different viewpoints. Then, a 3×3 spatial convolution is performed to expand the number of channels to a square of the upsampling factor of the network input image channels, i.e., a. 2 c is used to prepare for subpixel convolution operations. Finally, after subpixel convolution, pixels at the same position in the feature maps of each channel are arranged and fused in an orderly manner to achieve an upsampling of the spatial resolution of the light field image to a times that of the input light field image.

5. The method for generating a light field super-resolution network based on multi-view information interaction as described in claim 1, characterized in that: The discriminator works as follows: the super-resolution light field image output by the generator and the original high-resolution light field image are then fed into the discriminator. First, image features are extracted through 8 layers of decomposition kernel convolution, where all decomposition kernels are 3×3 in size. Except for the first decomposition kernel, the output channels of the odd-numbered decomposition kernel convolutional layers are twice the number of input channels, with a stride of 1. The output channels of the even-numbered decomposition kernel convolutional layers are the same as the number of input channels, but the stride is 2, so as to ensure that the size of the feature map is reduced to half of its original size each time the number of channels doubles. Except for the first decomposition kernel, all decomposition kernels are connected to a BN layer. Finally, the discriminant results are output after an average pooling layer and two 1×1 convolutions. The closer the discrimination result is to 1, the greater the probability that the discriminator considers the input image to be a high-resolution image. The closer the discrimination result is to 0, the greater the probability that the discriminator considers the input image to be a generated super-resolution image. This discrimination result will be fed back to the generator to guide the improvement of the generator.

Images