A multi-view information attention interaction network system for light field super-resolution

Abstract

The present invention discloses a multi-view information attention interaction network for light field super-resolution. A set of low-resolution light field images is used as the input of the network. The top branch includes a reshaping process, a feature extraction module, and four dual attention modules; the bottom branch includes a feature extraction module and four multi-view attention modules. The feature information extracted by these two branches is converged into a fusion reconstruction module. First, the global image features are used to update each single-view image feature so that each view feature can be integrated with the global view feature. Then, the feature is passed through an upsampling module and finally added to the bicubic upsampled image of the input image to obtain the final output image. The present invention uses a multi-view attention module to make the weight of the global view different, so that the information between different views can be better and more reasonably integrated. The present invention is better than other methods in terms of image detail and texture reconstruction effects, and the visual effect is closer to the ground truth image.

Description

Technical Field

[0001] The present invention relates to the field of light field imaging technology, and in particular to a multi-view information attention interaction network system for light field super-resolution. Background Art

[0002] In the digital and networked era, data flows rapidly, and cameras are rapidly entering the era of light field cameras. Light field (LF) imaging technology can record complete four-dimensional light field information. After four-dimensional image processing, it can realize functions such as stereo depth information extraction, free focus, and multi-view parallax image extraction. It has applications in many fields, including four-dimensional security monitoring, three-dimensional shape detection, facial recognition, and aerospace.

[0003] Although four-dimensional light field images provide both spatial and angular information, their low spatial resolution is a fundamental problem that hinders their potential. Therefore, to improve the spatial resolution of light field images, light field super-resolution (LFSR) algorithms have been widely studied. Recently, deep learning-based methods have been successfully applied to LFSR and achieved good performance. Due to occlusion and non-Lambertian reflectance in light fields, information from different viewpoints and channels has different importance. Existing learning-based LFSR methods treat the viewpoint features and spatial-channel features of light field images equally and cannot differentially utilize the information in light field images to further improve performance. Summary of the Invention

[0004] In view of this, in order to solve the above problems in the prior art, the present invention proposes a Multi-View Information Attention Interaction Network (MIAIN) for light field super-resolution (LFSR).

[0005] The present invention solves the above problems through the following technical means:

[0006] A multi-view information attention interaction network system for light field super-resolution, including a top branch, a bottom branch, a fusion reconstruction module and an upsampling module;

[0007] The top branch includes a reshaping process, a feature extraction module, and four dual attention modules connected in series; and the outputs of the four dual attention modules are cascaded;

[0008] The bottom branch comprises a feature extraction module and four multi-view attention modules connected in series; and the outputs of the four multi-view attention modules are cascaded;

[0009] A set of low-resolution light field images is used as the input of the network, and the feature information of the light field images is extracted through the top branch and the bottom branch respectively; the feature information extracted by these two branches is converged into the fusion reconstruction module. The fusion reconstruction module first uses the global image features to update the features of each single-view image, so that each view feature can be integrated with the global view feature. Then, it passes through the upsampling module and is finally added to the bicubic upsampled image of the input image to obtain the final output image.

[0010] Furthermore, the residual dense atrous spatial pyramid pooling module and residual block are used as the feature extraction module of the network; the convolution kernel expansion rates are selected as 1, 2 and 5; by cascading multiple atrous convolution layers, the neurons at the later stages can obtain larger and larger receptive fields; the receptive fields of atrous convolutions with expansion rates of 1, 2 and 5 are 3×3, 5×5 and 11×11 respectively, and the receptive field after cascading is 17×17; the input and output features of each layer are combined as the input of the next layer; the channel is compressed through a 1×1 convolution layer, and then activated by a rectified linear unit function after passing through a atrous convolution layer; finally, a 1×1 convolution layer is used at the end of the network to fuse the three extracted features; the features finally generated by the network can cover multiple scale ranges very densely; through shortcut connections, the network learns a residual mapping.

[0011] Furthermore, the input low-resolution sub-aperture image of the network is represented as I represents the light field image, I LR represents a low-resolution light field image, is a multidimensional real space; LR is low resolution; N is the total number of viewing angles of the light field; where N = U × V; U, V are the angular resolutions of the light field, U = V = 5; W, H are the sizes of the sub-aperture images, W = H = 32; the corresponding network output high-resolution sub-aperture image is expressed as Among them I SR stands for high-resolution light field image; SR is high resolution; α is the magnification of the network; in the bottom branch, the input I LR After the feature extraction module, the output features are C is the number of feature channels of the network input; the whole process is expressed as follows:

[0012] F views =H FEM (I LR ) (1-1)

[0013] Among them, F views is the extracted feature, H FEM Represents the action function of the feature extraction module, called shallow feature extractor; F viewsIt not only extracts the feature information of the single-view image, but also retains the correlation between views, which is used to facilitate the subsequent learning of the weights between the view features; in the top branch, input I LR After reshaping and then passing through the feature extraction module, the output features are F global It integrates the original correlation between views and extracts the feature information of the global view, which is used to focus on the hidden information inside the global image, making it easier to learn its weights in spatial position and channel.

[0014] Furthermore, the output features of the feature extraction module in the top branch are used as the input features F of the dual attention module. topin , the input feature F topin A one-dimensional global pooling operation along the vertical direction and a global pooling operation along the horizontal direction are performed respectively to encode each channel along the vertical direction and the horizontal direction to obtain a pair of direction-aware attention maps; the whole process is expressed as follows:

[0015]

[0016] Among them, H XGAP and H YGAP Represents the one-dimensional global pooling function along the horizontal and vertical directions respectively; F XGAP and F YGAP are the generated horizontal and vertical features respectively; the generated

[0017] Through these two transformations, not only does it have a global dependency on one spatial direction, but it also retains the precise position information of the other spatial direction. Then, the two generated features are concatenated and the intermediate feature is obtained through a shared 1×1 convolution layer and a rectified linear unit activation layer. Where γ represents the ratio of the compressed channel, which is used to reduce the amount of calculation; the whole process is expressed as follows:

[0018] F Mid =δ(H 1×1 ([F XGAP ,F YGAP ])) (1-3)

[0019] Where δ represents the rectified linear unit activation function, H 1×1 represents a shared 1×1 convolutional layer, [·] represents a cascade operation; F Mid Contains spatial information in the horizontal and vertical directions; then F is transformed along the spatial dimension Mid Split into two separate feature vectors and After a 1×1 convolution layer, the two feature vectors are expanded to the same number of channels as the input; then after a Sigmoid activation layer, we get and The whole process is shown as follows:

[0020]

[0021] Among them, F HS is the vertical output feature, F WS is the horizontal output feature, σ represents the Sigmoid activation function; finally, the feature vectors in the two directions are multiplied by the input, and the output is:

[0022] F topout =F topin *F HS *F WS (1-5)

[0023] Where, F topout is the learned spatial-channel attention feature, * represents the multiplication of the corresponding position elements in the two feature vectors; so far, the dual attention module integrates the spatial information in the horizontal and vertical directions by weighting them on the channel, achieving simultaneous attention to space and channel; at the end of the dual attention module, F topout Through two residual blocks, and then added to the input, we get the output of the dual attention module k represents the output of the kth dual attention module, k = 1, 2, 3, 4; the output features of the four dual attention modules are cascaded and then combined with the shallow features F extracted by the feature extraction module global Superposition, the final output of the top branch is:

[0024]

[0025] Among them, F Top is the final output feature of the top branch, These are the output features of the first, second, third, and fourth dual attention modules respectively; H 1×1 It is a 1×1 convolutional layer, which is used to reduce the number of feature channels after cascading from 4C to C.

[0026] Furthermore, the output features of the feature extraction module in the bottom branch are used as the input features F of the multi-view attention module. botin , first we need to transform the input feature F botin Dimensions, so Then the feature F botin Compressed by the maximum global pooling layer and the average global pooling layer respectively, we get and F GMP Global maximum feature, F GAP The global average feature is obtained. The two feature vectors then pass through a shared network layer, which contains two 1×1 convolutional layers and a rectified linear unit activation layer. The two 1×1 convolutional layers act as compression channels and expansion channels, respectively, with a compression ratio of θ to reduce the amount of computation. The two feature vectors are then summed and passed through a Sigmoid activation layer before being multiplied by the input feature and output. The entire process is shown below:

[0027]

[0028] in, Represents the 1×1 convolution layer weights of the compression channel, Represents the 1×1 convolution layer weight of the expansion channel, At the end of the multi-view attention module, the feature F MA After two residual blocks, the input F botin Superposition and summation are performed to obtain the final output of the module j=1,2,3,4; concatenate the output features of the four multi-view attention modules and combine them with the shallow features F extracted by the feature extraction module views The final output feature of the bottom branch is F Bottom , and transform its dimension; the specific process is as follows:

[0029]

[0030] in, are the output features of the first, second, third, and fourth multi-view attention modules, respectively. H 1×1 It is a 1×1 convolutional layer, which is used to reduce the number of feature channels after cascading from 4C to C.

[0031] Furthermore, we first use the global image feature F Top Update each single-view image feature F Bottom , so that each perspective feature can be integrated with the global perspective feature; the specific process is as follows: along the channel dimension, F Top With F Bottom Each view feature is cascaded to obtain the fusion feature Then the number of channels is compressed from 2C to C through a 1×1 convolution layer, and then the fused features are obtained through the rectified linear unit activation layer. F fus2 Each perspective feature is cascaded in the channel dimension to obtain the fusion feature Then, through the shared 1×1 convolution layer and the rectified linear unit activation layer, each view feature can share the information learned from each other; finally, reshape it back to the original dimension Input to the residual distillation module;

[0032] It consists of four residual distillation blocks. The features generated by the four residual distillation blocks are cascaded, and then compressed through a 1×1 convolution layer and a 3×3 convolution layer. Finally, the output feature F is superimposed and summed with the input. FRM ; The whole process is shown as follows:

[0033]

[0034] in, are the output features of the first, second, third, and fourth residual distillation blocks, respectively. The final fusion feature is used as the input feature of the residual distillation module;

[0035] i=1,2,3,4;H 3×3 is a 3×3 convolutional layer, H 1×1 It is a 1×1 convolutional layer, which is used to reduce the number of feature channels after cascading from 4C to C;

[0036] The distillation mechanism is used to gradually extract and process hierarchical features: the residual distillation block processes the input features Divided into two parts, one part passes through a 1×1 convolution layer and a rectified linear unit activation layer to compress the feature channel with a ratio of ε, and obtains The information distillation operation is realized; the other part is obtained after a 3×3 convolution layer, a shortcut connection branch and a rectified linear unit activation layer. The next information distillation is performed; after three repetitions, the features are compressed through a 3×3 convolution layer with a ratio of ε to obtain Then, together with the three previous Cascade, then pass through a 1×1 convolution layer to compress the channel, and finally superimpose and sum with the input through a shortcut connection to obtain the final output The whole process is shown as follows:

[0037]

[0038] Among them, F in is the input feature of the residual distillation block, These are the distillation features obtained from the first, second, third, and fourth information distillations respectively; The remaining features obtained from the first, second, and third information distillations; is the output feature of the i-th residual distillation block; in this way, useful information is gradually extracted and the SR performance is improved in an efficient way.

[0039] Furthermore, the feature F obtained from the fusion reconstruction module FRM Finally, it is fed into the upsampling module, which first passes through a 1×1 convolutional layer on F FRM Channel expansion is performed, and the ratio of channel expansion is related to the upsampling multiple, and the ratio size is equal to the square of the upsampling factor α; then it passes through the pixel mixing layer to generate a feature map of the target resolution αH×αW; finally, it passes through a 1×1 convolution layer to compress the number of channels to 1 and is combined with the input I after bicubic upsampling. LR The super-resolution reconstruction result I is obtained by superposition and summation. SR .

[0040] Compared with the prior art, the beneficial effects of the present invention include at least:

[0041] 1. The present invention uses a multi-view attention module to make the weights of global perspectives different, so as to better and more reasonably integrate information between different views.

[0042] 2. The present invention is better than other methods in terms of image details and texture reconstruction effects, and the visual effect is closer to the ground truth image.

[0043] 3. The present invention not only utilizes global information for super-resolution, but also assigns different weight values ​​to different perspectives, thereby achieving higher reconstruction accuracy with a relatively balanced PSNR distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 This is the overall architecture diagram of the multi-view information attention interaction network system MIAIN for light field super-resolution of the present invention;

[0046] Figure 2 Figure 1 is an architecture diagram of the feature extraction module of the present invention; (a) is a schematic diagram of the backbone network of the feature extraction module; (b) is the residual block; (c) is the RDASPP module;

[0047] Figure 3 It is an architectural diagram of the dual attention module of the present invention;

[0048] Figure 4It is the architecture diagram of the multi-view attention module of the present invention;

[0049] Figure 5 This is an overview diagram of the backbone network of the fusion reconstruction module of the present invention;

[0050] Figure 6 is an architecture diagram of the residual feature distillation module of the present invention;

[0051] Figure 7 is an architectural diagram of the residual feature distillation block of the present invention;

[0052] Figure 8 : This is the architecture diagram of the upsampling module of the present invention; (a) is an overview diagram of the upsampling module backbone network; (b) is a schematic diagram of Pixel Shuffle;

[0053] Figure 9 Comparison of the magnified areas and PSNR / SSIM values ​​of the central sub-aperture images reconstructed by different methods in the Cards scene in STFgantry at ×4. The upper left corner is the real surface image, and the red box is the local magnified area.

[0054] Figure 10 Comparison of the magnified areas and PSNR / SSIM values ​​of the central sub-aperture images reconstructed by different methods in the Bedroom scene in HCInew at ×4. The upper left corner shows the true surface image, and the red box shows the local magnified area.

[0055] Figure 11 Comparison of the magnified areas and PSNR / SSIM values ​​of the central sub-aperture images reconstructed by different methods in the herbs scene in HCInew at ×2. The upper left corner is the real surface image, and the red box is the local magnified area.

[0056] Figure 12 The PSNR distribution between different perspectives for the origami scene in HCInew at ×4 is shown. The average PSNR and standard deviation of the global perspective are also given. DETAILED DESCRIPTION

[0057] To make the above-mentioned objectives, features, and advantages of the present invention more clearly understood, the technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are also within the scope of protection of the present invention.

[0058] like Figure 1The present invention provides a multi-view information attention interaction network system for light field super-resolution, comprising a top branch, a bottom branch, a fusion reconstruction module and an upsampling module;

[0059] The top branch includes a reshaping process, a feature extraction module (FEM), and four dual attention modules (DAMs) connected in series; and the outputs of the four dual attention modules are cascaded;

[0060] The bottom branch includes a FEM and four multi-view attention modules (M-VAMs) connected in series, and the outputs of the four multi-view attention modules are cascaded;

[0061] The network takes a low-resolution light field image as input, extracting its features through the top and bottom branches. These features are then fed into the Fusion and Reconstruction Module (FRM), passed through the Upsampling Module (UPM), and finally added to the bicubic upsampled version of the input image to produce the final output image. MIAIN learns the difference between the reconstructed image and the high-resolution image, specifically the high-frequency details of the image.

[0062] 1.1 Feature Extraction Module

[0063] In convolutional neural networks, feature representations with rich spatial context information are beneficial to subsequent information interaction. Therefore, a large receptive field with dense pixel sampling rate is required to extract hierarchical features. Figure 2As shown, the present invention utilizes the RDASPP (Residual Dense Atrous Spatial Pyramid Pooling) module and the residual block (ResidualBlock, ResBlock) as the main feature extraction modules of the network. The design of RDASPP is inspired by the Densegaspp network proposed by Yang et al., which uses dense connections to combine the outputs of each dilated convolution layer (DilatedConvolution). The dilated convolution layer can expand the receptive field without increasing the number of convolution kernel parameters; in order to eliminate the "gridding" problem caused by dilated convolution, referring to the design scheme proposed by Wang et al., the convolution kernel expansion rates selected by the present invention are 1, 2, and 5. By cascading multiple dilated convolution layers, the later neurons can obtain larger and larger receptive fields; the receptive fields of dilated convolutions with expansion rates of 1, 2, and 5 are 3×3, 5×5, and 11×11, respectively, and the receptive field after cascading is 17×17. The input and output features of each layer are combined as the input of the next layer; the channel is compressed through a 1×1 convolution layer, and then activated by the Leaky-ReLU (LReLU) function after passing through a dilated convolution layer; finally, a 1×1 convolution layer is used at the end of the network to fuse the three extracted features; the features finally generated by the network can cover multiple scale ranges very densely. Through the shortcut connection method, the network learns a residual mapping, and the residual network can effectively solve the network degradation problem, such as Figure 2 (b) and (c) both introduce the concept of residual.

[0064] The input low-resolution SAIs of the network can be expressed as I represents the light field image, I LR represents a low-resolution light field image, is a multidimensional real space; LR is low resolution; N is the total number of viewing angles of the light field; where N = U × V; U, V are the angular resolutions of the light field, U = V = 5; W, H are the sizes of the sub-aperture images, W = H = 32; the corresponding network output high-resolution sub-aperture image is expressed as Among them I SR stands for high-resolution light field image; SR is high resolution; α is the magnification of the network; in the bottom branch, the input I LR After the feature extraction module, the output features are C is the number of feature channels of the network input; the whole process is expressed as follows:

[0065] F views =H FEM (I LR ) (1-1)

[0066] Among them, F views is the extracted feature, H FEM Represents the action function of FEM, which can also be called shallow feature extractor. views It not only extracts the feature information of the single-view image, but also preserves the correlation between views, which facilitates the subsequent learning of the weights between the view features. LR After reshaping and then passing through FEM, the output features are F global The original correlation between views is integrated and the feature information of the global view is extracted. The purpose is to focus on the hidden information inside the global image and facilitate the subsequent learning of its weights in spatial position and channel.

[0067] 1.2 Dual Attention Module

[0068] In convolutional neural networks, the features extracted by the shallow feature extraction module are input into the subsequent multi-layer network. The multi-layer network treats each shallow feature indiscriminately, which obviously limits the network's representational capabilities. Therefore, it is necessary to assign different weights to different shallow features so that the network can select more critical and important information, thereby improving network performance. Therefore, the present invention introduces an attention mechanism and designs the DAM with reference to the Coordinate Attention (CA) proposed by Hou et al. to help the network select more important information in the channel domain and spatial domain.

[0069] like Figure 3 The output features of the feature extraction module in the top branch are used as the input features F of the dual attention module. topin , the input feature F topin A one-dimensional global pooling operation along the vertical direction and a global pooling operation along the horizontal direction are performed respectively. The purpose is to encode each channel along the vertical direction and the horizontal direction to obtain a pair of direction-aware attention maps. The whole process is expressed as follows:

[0070]

[0071] Among them, H XGAP and H YGAP Represents the one-dimensional global pooling function along the horizontal and vertical directions respectively; F XGAP and F YGAP are the generated horizontal and vertical features respectively; the generated

[0072] Through these two transformations, we not only have global dependence on one spatial direction, but also retain the precise position information of the other spatial direction. Then, after the two generated features are cascaded, a shared 1×1 convolution layer and an LReLU activation layer are used to obtain the intermediate feature. Where γ represents the ratio of the compressed channel, the purpose of which is to reduce the amount of calculation. The whole process is expressed as follows:

[0073] F Mid =δ(H 1×1 ([F XGAP ,F YGAP ])) (1-3)

[0074] Among them, δ represents the LReLU activation function, H 1×1 represents a shared 1×1 convolutional layer, and [·] represents a cascade operation. Mid Contains spatial information in the horizontal and vertical directions. Then, F is transformed along the spatial dimension Mid Split into two separate feature vectors and After a 1×1 convolution layer, the two feature vectors are expanded to the same number of channels as the input; then after a Sigmoid activation layer, we get and The whole process is shown as follows:

[0075]

[0076] Among them, F HS is the vertical output feature, F WS is the horizontal output feature, and σ represents the Sigmoid activation function. Finally, the feature vectors in both directions are multiplied by the input, and the output is:

[0077] F topout =F topin *F HS *F WS (1-5)

[0078] Where, F topout is the learned spatial-channel attention feature, * represents the multiplication of the corresponding position elements in the two feature vectors. So far, this module has integrated the horizontal and vertical spatial information by weighting them on the channel, thus achieving the simultaneous attention to space and channel. At the end of DAM, F topout Through two residual blocks, and then added to the input, the output of DAM is obtained k represents the kth DAM output, k = 1, 2, 3, 4. The output features of the four DAMs are cascaded and then combined with the shallow features F extracted by FEM. globalSuperposition, the final output of the top branch is:

[0079]

[0080] Among them, F Top is the final output feature of the top-level branch, These are the output features of the first, second, third, and fourth dual attention modules respectively; H 1×1 It is a 1×1 convolution layer, the purpose of which is to reduce the number of feature channels after cascading from 4C to C.

[0081] 1.3 Multi-view Attention Module

[0082] In previous light field super-resolution work, researchers have treated all viewpoint features equally, which limits information exchange between different viewpoints. Therefore, the present invention designs M-VAM to adaptively select features from different viewpoints, thereby helping the network improve its representation capabilities.

[0083] like Figure 4 The output features of the feature extraction module in the bottom branch are used as the input features F of the multi-view attention module. botin , first we need to transform the input feature F botin Dimensions, so Then the feature F botin Compressed by the maximum global pooling layer and the average global pooling layer respectively, we get and F GMP Global maximum feature, F GAP The two feature vectors are then passed through a shared network layer consisting of two 1×1 convolutional layers and an LReLU activation layer. The two 1×1 convolutional layers act to compress and expand the channel, respectively, with a compression ratio of θ to reduce computational complexity. The two feature vectors are then summed, passed through a Sigmoid activation layer, and finally multiplied by the input feature before being output. The entire process is shown below:

[0084]

[0085] in, Represents the 1×1 convolution layer weights of the compression channel, Represents the 1×1 convolution layer weight of the expansion channel, At the end of M-VAM, feature F MA After two residual blocks, the input F botin Superposition and summation are performed to obtain the final output of the module j=1,2,3,4. The output features of the four M-VAMs are cascaded and then combined with the shallow features F extracted by FEM.views The final output feature of the bottom branch is F Bottom , and transform its dimension. The specific process is as follows:

[0086]

[0087] in, are the output features of the first, second, third, and fourth multi-view attention modules, respectively. H 1×1 It is a 1×1 convolution layer, the purpose of which is to reduce the number of feature channels after cascading from 4C to C.

[0088] 1.4 Fusion and Reconstruction Module

[0089] In order to achieve better super-resolution reconstruction effect, it is necessary to transform the feature F of the global image obtained by the top branch into Top and the feature F between image view relations obtained by the bottom branch Bottom Therefore, the present invention designs FRM, such as Figure 5 As shown. First, use the global image feature F Top Update each single-view image feature F Bottom , so that each perspective feature can be integrated with the global perspective feature. The specific process is as follows: along the channel dimension, F Top With F Bottom Each view feature is cascaded to obtain the fusion feature Then the number of channels is compressed from 2C to C through a 1×1 convolution layer, and then the fusion feature is obtained through the LReLU activation layer. F fus2 Each perspective feature is cascaded in the channel dimension to obtain the fusion feature Then, through the shared 1×1 convolution layer and LReLU activation layer, the purpose is to allow each view feature to share the information learned from each other. Finally, reshape it back to the original dimension Input to RFDM (Residual Feature Distillation Module).

[0090] RFDM backbone diagram Figure 6 As shown in the figure, it consists of four RFDB (Residual Feature Distillation Block). The features generated by the four RFDBs are cascaded, and then compressed through a 1×1 convolution layer and a 3×3 convolution layer. Finally, the output feature F is superimposed and summed with the input. FRM The whole process is shown as follows:

[0091]

[0092] in, are the output features of the first, second, third, and fourth residual distillation blocks, respectively. The final fusion feature is used as the input feature of the residual distillation module; i=1,2,3,4. H 3×3 is a 3×3 convolutional layer, H 1×1 It is a 1×1 convolution layer, the purpose of which is to reduce the number of feature channels after cascading from 4C to C.

[0093] Based on Liu et al., the present invention designs RFDB as Figure 7 As shown in Figure 2. By using a distillation mechanism to gradually extract and process hierarchical features, superior stochastic resonance performance can be achieved with fewer parameters and lower computational cost. Specifically, RFDB Divided into two parts, one part passes through a 1×1 convolution layer and an LReLU activation layer to compress the feature channel with a ratio of ε, and obtains The other part passes through a 3×3 convolution layer, a quick connection branch and an LReLU activation layer to obtain The next information distillation is carried out. After three repetitions, the features are compressed through a 3×3 convolution layer with a ratio of ε to obtain Then, together with the three previous Cascade, then pass through a 1×1 convolution layer to compress the channel, and finally superimpose and sum with the input through a shortcut connection to obtain the final output The whole process is shown as follows:

[0094]

[0095] Among them, F in is the input feature of the residual distillation block, These are the distillation features obtained from the first, second, third, and fourth information distillations respectively; The remaining features obtained from the first, second, and third information distillations; is the output feature of the i-th residual distillation block; in this way, useful information can be gradually extracted and the SR performance can be improved in an efficient way.

[0096] 1.5 Upsampling Module

[0097] The feature F obtained from the fusion reconstruction module FRM Finally, it is fed into the upsampling module, such as Figure 8 (a). First, a 1×1 convolutional layer is used to transform F FRMChannel expansion is performed. The ratio of channel expansion is related to the upsampling multiple, and the ratio is equal to the square of the upsampling factor α. Then it passes through the Pixel Shuffle layer, such as Figure 8 As shown in (b), a feature map of the target resolution αH×αW is generated. Finally, a 1×1 convolution layer is used to compress the number of channels to 1 and then compare it with the input I after bicubic upsampling. LR The super-resolution reconstruction result I is obtained by superposition and summation. SR .

[0098] 2 Experimental results and analysis

[0099] 2.1 Experimental Details

[0100] The experiment of the present invention used two public synthetic light field datasets, HCInew and HCIold, and three public real-world light field datasets, EPFL, INRIA, and STFgantry. The EPFL and INRIA datasets were taken with Lytro cameras, with small baselines and small parallax; the HCInew and HCIold datasets were simulated by computer software; and the STFgantry dataset was taken with a camera array, with a large baseline and large parallax. Training synthetic images and real-world images together ensures the diversity of the dataset for robust inference. Specifically, a total of 144 light field images, including 30 synthetic light field images and 114 real-world light field images, were used for training; and a total of 23 light field images, including 6 synthetic scenes and 17 real-world scenes, were used for testing. Table 4-1 lists the number of training and test scenes for each dataset.

[0101] Table 4-1 Datasets used in the experiment. The first two are synthetic datasets, and the last three are real datasets.

[0102] Dataset Number of training sessions Number of tests HCInew 20 4 HCIold 10 2 EPFL 70 10 INRIA 35 5 STFgantry 9 2 Total 144 23

[0103] All light field images in these datasets have an angular resolution of 5×5. During the training phase, each light field image is cropped into high-resolution image patches of 64×64 (α=2) and 128×128 (α=4), and downsampled to low-resolution image patches of 32×32 using bicubic interpolation. This method yields 43,773 HR-LR training pairs (α=2) and 9,044 HR-LR training pairs (α=4), respectively. In addition, these high-resolution-low-resolution image pairs are randomly flipped and rotated 90 degrees in both spatial and angular directions for data augmentation. Note that the spatial and angular dimensions need to be jointly flipped or rotated to preserve the light field structure.

[0104] Because the human eye is more sensitive to brightness information, in our experiments, we converted the light field images to the YCbCr color space and calculated PSNR and SSIM only on the Y channel to evaluate model performance. Bicubic upsampling was used directly on the other two channels. To obtain a metric score for a dataset containing M scenes, we calculated the metric scores for each of the N viewpoints, then averaged the scores for the N viewpoints to obtain the score for that scene, and finally averaged the scores for the M scenes to obtain the metric score for the dataset.

[0105] The network model was implemented using PyTorch on a computer with two NVIDIA RTX-2080Ti graphics processors. The computer's CPU was an i7-9700K, the system was Ubuntu 18.04, 64GB of RAM, and the acceleration tools were Cuda 10.1 and CuDNN 7.

[0106] In the network of the present invention, for the input low-resolution light field I LR The number of channels is set to C = 64, N = 25, α = 2 or 4, the compression ratio γ = θ = 4 in DAM and M-VAM, and the compression ratio ε = 2 in RFDB. The network weights are initialized using the Kaiming method and optimized using the Adam gradient optimization algorithm. When α = 2, the training batch size is 8; when α = 4, the training batch size is 4. The initial learning rate is 4×10 -4 , the learning rate is reduced by half every 15 cycles, and the training is stopped at 100 cycles. The network proposed in this invention super-resolves each sub-aperture image in SAIs, and reconstructs the result of super-resolution by minimizing I SR and the corresponding ground truth image I HR The difference between them is optimized. The L1 loss function is applied for training as follows:

[0107]

[0108] 2.2 Comparison with Existing Methods

[0109] In order to evaluate the performance of the model of the present invention, MIAIN of the present invention is compared with several state-of-the-art methods. Three CNN-based SISR methods, namely VDSR, EDSR, and RCAN, and seven CNN-based LFSR methods, namely LFSSR, resLF, LF-ATO, LF-InterNet, MEG-Net, LF-DFNet, and DPT, are included; the bicubic interpolation results are used as the benchmark. In order to make a fair comparison, all CNN-based methods of the present invention are retrained on the same training dataset. In order to make a comprehensive comparison, the present invention gives the quantitative results of ×2 and ×4 light field super-resolution reconstruction. Table 4-2 lists the comparison results of PSNR and SSIM on the test sets of the five datasets, and also provides the average PSNR and SSIM results of all test scenes. The best results in the table are highlighted in black bold, and the suboptimal results are highlighted with underlines.

[0110] Table 4-2 shows the SSIM results obtained using different methods for LFSRs with a resolution of ×2 or ×4. The best results are highlighted in bold black, and the suboptimal results are underlined.

[0111]

[0112] Table 4-2 shows that the proposed MIAIN achieves excellent results in both the ×2 and ×4 scenarios. Compared with the single-image super-resolution methods VDSR, EDSR, and RCAN, SISR super-resolves each sub-aperture image separately, ignoring the angular correlation of SAIs; MIAIN emphasizes the correlation between viewpoints, resulting in a significant performance improvement. Compared with other CNN-based LFSR methods, the proposed method ranks first in average PSNR and SSIM values ​​across the five test sets. In particular, in the ×4 scenario, the proposed method outperforms other methods on each test set, with average PSNRs 0.30dB and 0.22dB higher than the latest DPT and LF-DFNet, and average SSIMs 0.0035 and 0.0023 higher. This is because other methods distribute the contribution of global viewpoints equally, while the proposed method utilizes a multi-view attention module to vary the weights of global viewpoints, enabling better and more reasonable fusion of information between different views.

[0113] Figure 9 、 10The following table shows some representative LFSR visualization results of different methods under the condition of ×4. As can be seen from the enlarged area of ​​the figure, SISR cannot reliably reconstruct the missing details. In contrast, LFSR significantly improves the reconstruction effect by utilizing the angle information. Compared with the state-of-the-art SISR and other LFSR methods, the results of the proposed MIAIN have more high-frequency details and fewer artifacts, such as Figure 4-9 The letters and Figure 4-10 Strips on the wall.

[0114] In the ×2 case, while our method does not outperform the state-of-the-art DPT model on the small-disparity datasets EPFL and INRIA, it achieves average PSNR improvements of 0.39dB, 0.40dB, and 0.62dB higher on the medium-disparity datasets HCInew and HCIold, and the large-disparity dataset STFgantry, respectively. This is because these three datasets are either synthetically rendered or captured by a mobile camera mounted on a gantry, resulting in more complex structures and greater disparity variation than the Lytro dataset. The multi-view attention module and dual-attention module effectively extract complementary information between different views and assign different weights, effectively handling these complex scenes. In contrast, on the smaller Lytro dataset, the contributions of different viewpoints to the global image are similar due to the minimal variation between adjacent sub-aperture images, thus underperforming our method. The final results show that while our method lags slightly behind the state-of-the-art DPT and LF-DFNet on the EPFL and INRIA datasets, it still outperforms other methods and maintains first place on average across all five datasets. Figure 11 The representative LFSR visualization results of different methods are shown in the figure. The proposed method is better than other methods in terms of image details and texture reconstruction, and the visual effect is closer to the ground truth image.

[0115] Since the ultimate goal of LFSR is to super-resolve the sub-aperture images of each viewpoint, this paper studies the reconstruction accuracy of different methods under different viewpoints. This paper selects the origami scene in the HCInew dataset and calculates the PSNR value of each sub-aperture image in the ×4 case. Because SISR does not use complementary information to super-resolve each SAIs independently, we only compare the PSNR value of each SAIs in the learning-based LFSR method, such as Figure 12As shown in the figure. It can be seen from the results in the figure that the method of the present invention has a good super-resolution effect for each perspective. Compared with other LFSR methods, the average PSNR value of the present invention in this scenario is 0.27dB and 0.33dB higher than the most advanced DPT and DFnet. Because resLF only uses part of the perspective for super-resolution, the distribution of PSNR is uneven, and the standard deviation reaches 0.1781. The method of the present invention not only uses global information for super-resolution, but also assigns different weight values ​​to different perspectives, thereby achieving higher reconstruction accuracy with a relatively balanced PSNR distribution.

[0116] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A multi-view information attention interaction network system for light field super-resolution, characterized by: Includes top branch, bottom branch, fusion reconstruction module and upsampling module; The top branch includes a reshaping process, a feature extraction module, and four dual attention modules connected in series; and the outputs of the four dual attention modules are cascaded; The bottom branch contains a feature extraction module and four multi-view attention modules connected in series; And concatenate the outputs of the four multi-view attention modules; A set of low-resolution light field images is used as the input of the network. The feature information of the light field images is extracted through the top branch and the bottom branch respectively. The feature information extracted by these two branches is converged into the fusion reconstruction module. The fusion reconstruction module first uses the global image features to update the features of each single-view image, so that the features of each view can be integrated with the global view features. Then, the fusion reconstruction module passes through the upsampling module and finally adds the bicubic upsampled image of the input image to obtain the final output image. The residual dense atrous spatial pyramid pooling module and residual blocks are used as the network's feature extraction module. The convolution kernel dilation rates are selected to be 1, 2, and 5. By cascading multiple atrous convolution layers, the later neurons can obtain increasingly larger receptive fields. The receptive fields of atrous convolutions with dilation rates of 1, 2, and 5 are 3×3, 5×5, and 11×11, respectively, and the receptive field after cascading is 17×17. The input and output features of each layer are combined as the input of the next layer. The channel is compressed through a 1×1 convolution layer, and then activated by a rectified linear unit function after passing through a dilated convolution layer. Finally, a 1×1 convolution layer is used at the end of the network to fuse the three extracted features. The features ultimately generated by the network can cover multiple scale ranges very densely. Through shortcut connections, the network learns a residual mapping. The input low-resolution sub-aperture image of the network is represented as I represents the light field image, I LR represents a low-resolution light field image, is a multidimensional real space; LR is low resolution; N is the total number of viewing angles of the light field; where N = U × V; U, V are the angular resolutions of the light field, U = V = 5; W, H are the sizes of the sub-aperture images, W = H = 32; the corresponding network output high-resolution sub-aperture image is expressed as Among them I SR stands for high-resolution light field image; SR is high resolution; α is the magnification of the network; in the bottom branch, the input I LR After the feature extraction module, the output features are C is the number of feature channels of the network input; the whole process is expressed as follows: F views =H FEM (I LR ) (1-1) Among them, F views is the extracted feature, H FEM Represents the action function of the feature extraction module, called shallow feature extractor; F views It not only extracts the feature information of the single-view image, but also retains the correlation between views, which is used to facilitate the subsequent learning of the weights between the view features; in the top branch, input I LR After reshaping and then passing through the feature extraction module, the output features are F global It integrates the original correlation between views and extracts the feature information of the global view, which is used to focus on the hidden information within the global image, facilitating the subsequent learning of its weight in spatial position and channel. The output features of the feature extraction module in the top branch are used as the input features F of the dual attention module topin , the input feature F topin A one-dimensional global pooling operation along the vertical direction and a global pooling operation along the horizontal direction are performed respectively to encode each channel along the vertical direction and the horizontal direction to obtain a pair of direction-aware attention maps; the whole process is expressed as follows: Among them, H XGAP and H YGAP Represents the one-dimensional global pooling function along the horizontal and vertical directions respectively; F XGAP and F YGAP are the generated horizontal and vertical features respectively; the generated Through these two transformations, not only does it have a global dependency on one spatial direction, but it also retains the precise position information of the other spatial direction. Then, the two generated features are concatenated and the intermediate feature is obtained through a shared 1×1 convolution layer and a rectified linear unit activation layer. Where γ represents the ratio of the compressed channel, which is used to reduce the amount of calculation; the whole process is expressed as follows: F Mid =δ(H 1×1 ([F XGAP ,F YGAP ])) (1-3) Where δ represents the rectified linear unit activation function, H 1×1 represents a shared 1×1 convolutional layer, [·] represents a cascade operation; F Mid Contains spatial information in the horizontal and vertical directions; then F is transformed along the spatial dimension Mid Split into two separate feature vectors and After a 1×1 convolution layer, the two feature vectors are expanded to the same number of channels as the input; then after a Sigmoid activation layer, we get and The whole process is shown as follows: Among them, F HS is the vertical output feature, F WS is the horizontal output feature, σ represents the Sigmoid activation function; finally, the feature vectors in the two directions are multiplied by the input, and the output is: F topout =F topin *F HS *F WS (1-5) Where, F topout is the learned spatial-channel attention feature, * represents the multiplication of the corresponding position elements in the two feature vectors; so far, the dual attention module integrates the spatial information in the horizontal and vertical directions by weighting them on the channel, achieving simultaneous attention to space and channel; at the end of the dual attention module, F topout Through two residual blocks, and then added to the input, we get the output of the dual attention module k represents the output of the kth dual attention module, k = 1, 2, 3, 4; the output features of the four dual attention modules are cascaded and then combined with the shallow features F extracted by the feature extraction module global Superposition, the final output of the top branch is: Among them, F Top is the final output feature of the top-level branch, These are the output features of the first, second, third, and fourth dual attention modules respectively; H 1×1 It is a 1×1 convolutional layer, which is used to reduce the number of feature channels after cascading from 4C to C; The output features of the feature extraction module in the bottom branch are used as the input features F of the multi-view attention module botin , first we need to transform the input feature F botin Dimensions, so Then the feature F botin Compressed by the maximum global pooling layer and the average global pooling layer respectively, we get and F GMP Global maximum feature, F GAP The global average feature is obtained. The two feature vectors then pass through a shared network layer, which contains two 1×1 convolutional layers and a rectified linear unit activation layer. The two 1×1 convolutional layers act as compression channels and expansion channels, respectively, with a compression ratio of θ to reduce the amount of computation. The two feature vectors are then summed and passed through a Sigmoid activation layer before being multiplied by the input feature and output. The entire process is shown below: in, Represents the 1×1 convolution layer weights of the compression channel, Represents the 1×1 convolution layer weight of the expansion channel, At the end of the multi-view attention module, the feature F MA After two residual blocks, the input F botin Superposition and summation are performed to obtain the final output of the module The output features of the four multi-view attention modules are cascaded and then combined with the shallow features F extracted by the feature extraction module. views The final output feature of the bottom branch is F Bottom , and transform its dimension; the specific process is as follows: in, are the output features of the first, second, third, and fourth multi-view attention modules, respectively. H 1×1 It is a 1×1 convolutional layer, which is used to reduce the number of feature channels after cascading from 4C to C; First, we use the global image feature F Top Update each single-view image feature F Bottom , so that each perspective feature can be integrated with the global perspective feature; the specific process is as follows: along the channel dimension, F Top With F Bottom Each view feature is cascaded to obtain the fusion feature Then the number of channels is compressed from 2C to C through a 1×1 convolution layer, and then the fused features are obtained through the rectified linear unit activation layer. F fus2 Each perspective feature is cascaded in the channel dimension to obtain the fusion feature Then, through the shared 1×1 convolution layer and the rectified linear unit activation layer, each view feature can share the information learned from each other; finally, reshape it back to the original dimension Input to the residual distillation module; It consists of four residual distillation blocks. The features generated by the four residual distillation blocks are cascaded, and then compressed through a 1×1 convolution layer and a 3×3 convolution layer. Finally, the output feature F is superimposed and summed with the input. FRM ; The whole process is shown as follows: in, are the output features of the first, second, third, and fourth residual distillation blocks, respectively. The final fusion feature is used as the input feature of the residual distillation module; H 3×3 is a 3×3 convolutional layer, H 1×1 It is a 1×1 convolutional layer, which is used to reduce the number of feature channels after cascading from 4C to C; A distillation mechanism is used to gradually extract and process hierarchical features: the residual distillation block processes the input features Divided into two parts, one part passes through a 1×1 convolution layer and a rectified linear unit activation layer to compress the feature channel with a ratio of ε, and obtains The information distillation operation is realized; the other part is obtained after a 3×3 convolution layer, a shortcut connection branch and a rectified linear unit activation layer. The next information distillation is performed; after three repetitions, the features are compressed through a 3×3 convolution layer with a ratio of ε to obtain Then, together with the three previous Cascade, then pass through a 1×1 convolution layer to compress the channel, and finally superimpose and sum with the input through a shortcut connection to obtain the final output The whole process is shown as follows: Among them, F in is the input feature of the residual distillation block, These are the distillation features obtained from the first, second, third, and fourth information distillations respectively; The remaining features obtained from the first, second, and third information distillations; is the output feature of the i-th residual distillation block; in this way, useful information is gradually extracted and the SR performance is improved in an efficient way; The feature F obtained from the fusion reconstruction module FRM Finally, it is fed into the upsampling module, which first passes through a 1×1 convolutional layer on F FRM Channel expansion is performed, and the ratio of channel expansion is related to the upsampling multiple, and the ratio size is equal to the square of the upsampling factor α; then it passes through the pixel mixing layer to generate a feature map of the target resolution αH×αW; finally, it passes through a 1×1 convolution layer to compress the number of channels to 1 and is combined with the input I after bicubic upsampling. LR The super-resolution reconstruction result I is obtained by superposition and summation. SR .

Images