Light field image super-resolution reconstruction method and device based on state model

Abstract

The invention discloses a light field image super-resolution reconstruction method and device based on a state model, and the method comprises the steps: constructing a shallow feature extraction module which is used for extracting shallow features from the sub-aperture image form of an input to-be-reconstructed light field image; constructing a deep feature extraction unit by using the state space model; a light field space angle feature reconstruction module is constructed based on the deep feature extraction unit and used for reconstructing light field space angle features for the shallow features; a light field structure feature reconstruction module is constructed based on the deep feature extraction unit and used for reconstructing light field structure features for the shallow features; and the high-resolution image reconstruction module is used for carrying out hierarchical feature fusion and up-sampling on the shallow layer features, the light field empty angle features and the light field structure features to obtain a reconstructed high-resolution light field image. According to the invention, the dynamic characteristics of the state space model are utilized, so that the global space angle information characterization capability and the detail reconstruction capability of the light field image super-resolution method are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a state model-based light field image super-resolution reconstruction method and device. Background Art

[0002] Image super-resolution (SR) is one of the key tasks in computer vision and image processing, which aims to reconstruct high-quality high-resolution (HR) images from existing low-resolution (LR) images. Light field images captured by light field cameras have the ability to record light intensity and direction, which makes them show great potential in applications such as depth estimation, refocusing, and saliency detection. However, these cameras often sacrifice the spatial resolution of each sub-aperture image (SAI) when capturing multiple perspectives, which limits their effectiveness in high-quality visual applications. To solve this problem, Light Field Image Super-Resolution (LFSR) technology came into being, aiming to improve the spatial resolution of low-resolution light field images to better support subsequent visual analysis and processing.

[0003] Compared with ordinary image super-resolution technology, the difficulty of light field image super-resolution technology lies in how to make full use of the inherent spatial and angular information in light field images to improve the reconstruction accuracy. In recent years, in the field of LFSR, technologies based on convolutional neural networks (CNN) and transformers have achieved certain results. These technologies significantly improve the spatial resolution of light field images by improving the resolution of each SAI and combining adjacent high-resolution SAIs for high-resolution light field image reconstruction compared to traditional methods. However, this method still ignores the rich angular information between different SAIs during super-resolution reconstruction, resulting in less than optimal performance. At the same time, since the convolution operation extracts local features within a small neighborhood by using a fixed sliding window, the CNN-based LFSR method can only process the local empty angle features of light field images; the Transformer-based LFSR method can explore the non-local empty angle features of light field images, but the quadratic computational complexity of self-attention calculation limits its efficient and comprehensive exploration of the intrinsic structural characteristics of light field images.

[0004] As an emerging sequence modeling architecture, the state-space model has gradually shown its feasibility and superiority in intelligent vision tasks. How to use the dynamic characteristics of the state-space model to effectively learn the long-range dependencies of the intrinsic structure of light field images, and fully interact with the spatial and angular information in the light field images to improve the reconstruction quality of low-resolution light field images is a problem worth exploring. Summary of the invention

[0005] The main purpose of the present invention is to overcome the problems that the existing light field image super-resolution methods cannot fully exchange the empty angle information of the light field image and cannot effectively explore the intrinsic structural characteristics of the light field image, and propose a light field image super-resolution reconstruction method and device based on a state model. By utilizing the dynamic characteristics of the state space model, the global empty angle information representation ability and detail reconstruction ability of the light field image super-resolution method are significantly improved.

[0006] The present invention adopts the following technical solution:

[0007] A light field image super-resolution reconstruction method based on a state model, comprising:

[0008] Constructing a shallow feature extraction module for extracting shallow features from a sub-aperture image of an input light field image to be reconstructed;

[0009] Using state-space models to build deep feature extraction units;

[0010] Constructing a light field empty angle feature reconstruction module based on the deep feature extraction unit, for reconstructing the light field empty angle feature for the shallow feature;

[0011] Constructing a light field structure feature reconstruction module based on the deep feature extraction unit, for reconstructing light field structure features for the shallow features;

[0012] A high-resolution image reconstruction module is constructed to perform hierarchical feature fusion and up-sampling on the shallow features, the light field empty angle features and the light field structure features to obtain a reconstructed high-resolution light field image.

[0013] The shallow feature extraction module constructed is expressed as follows:

[0014] ;

[0015] ;

[0016] in, represents the sub-aperture image form of the light field image to be reconstructed, Represents shallow features, represents the shallow feature extraction module, Indicates that the convolution kernel size is The convolutional layer, represents the convolutional layer function for channel upsampling, Represents the LReLU activation function.

[0017] The deep feature extraction unit constructed using the state space model is as follows:

[0018] ;

[0019] ; ;

[0020] in, Represents shallow features of light field images, Represents the local intermediate features of the light field image, represents the depth features of the light field image, represents the deep feature extraction unit, The function representing the normalization of the layer, express The convolutional layer, The function representing the channel attention, , represents a learnable residual connection, represents the selective state-space model.

[0021] The Selectivity State Space Model The expression is as follows:

[0022] ;

[0023] in, represents the weighted features of the branch output where the state space model is located, represents the supplementary features of the non-state-space model branch output, Indicates that the feature , The final refined features obtained by fusion, represents a linear mapping layer, represents depthwise separable convolution, represents the activation function, represents the state space model, represents the Hadamard dot product.

[0024] The state space model The calculation process is as follows:

[0025] ;

[0026] in, , Respectively represent the length The input sequence and output sequence of Indicates the length is The learnable hidden state space of represents the hidden state space at the previous moment, , and represents the discretized parameter matrix, and The value is usually 64. Represents a real number.

[0027] A light field empty angle feature reconstruction module is constructed based on the deep feature extraction unit, specifically including the following:

[0028] Constructing spatial feature extraction unit , which includes a matrix dimension conversion unit and deep feature extraction unit The cascade structure is as follows:

[0029] ;

[0030] in, represents the shallow features, represents a matrix dimension transformation operation to transform the shallow features into sub-aperture image features, represents the deep feature extraction unit;

[0031] Constructing the angle feature extraction unit , which includes a matrix dimension conversion unit and deep feature extraction unit The cascade structure is described as follows:

[0032] ;

[0033] in, Describing the sub-aperture image characteristics, representing a matrix dimension conversion operation to convert the sub-aperture image features into macro-pixel image features;

[0034] The spatial feature extraction unit and the angle feature extraction unit are integrated to reconstruct the light field spatial angle feature, which is expressed as follows:

[0035] ;

[0036] ;

[0037] in, Represents the function of the light field empty angle feature reconstruction module, Represents the reconstructed light field empty angle feature.

[0038] A light field structure feature reconstruction module is constructed based on the deep feature extraction unit, which specifically includes the following:

[0039] Constructing horizontal light field structure feature extraction unit , which uses a matrix dimension transformation unit and deep feature extraction unit Cascaded, as shown below:

[0040] ;

[0041] in, represents the reconstructed light field empty angle feature, Represents the matrix dimension transformation operation to transform the light field empty angle feature Transformed into horizontal epipolar plane image features , represents the deep feature extraction unit;

[0042] Constructing a vertical light field structure feature extraction unit , which uses a matrix dimension transformation unit and deep feature extraction unit The cascade structure is as follows:

[0043] ;

[0044] in, represents the matrix dimension transformation operation to transform the horizontal epipolar plane image features Transformed into vertical epipolar plane image features;

[0045] The horizontal light field structure feature extraction unit and the vertical light field structure feature extraction unit Alternately cascade three times and reconstruct the light field empty angle feature by residual connection The light field structure feature reconstruction module is constructed as follows:

[0046] ;

[0047] ;

[0048] in, represents the light field structure feature reconstruction module, Represents the reconstructed light field structure features.

[0049] Constructing a high-resolution image reconstruction module for hierarchical feature fusion and upsampling of the shallow features, the light field empty angle features, and the light field structure features to obtain a reconstructed high-resolution light field image, specifically including the following:

[0050] Construct a hierarchical feature fusion unit, which converts the input shallow features, the reconstructed light field empty angle features, and the reconstructed light field structure features into light field fusion features , which is expressed as follows:

[0051] ;

[0052] ;

[0053] in, Indicates that the convolution kernel size is The convolutional layer, represents the channel dimension concatenation operation, , , Respectively represent shallow features, reconstructed light field empty angle features, and reconstructed light field structure features;

[0054] Construct a fusion feature upsampling unit, The function representing the fusion feature upsampling unit is expressed as follows:

[0055] ;

[0056] in, The scale factor is or Sub-pixel convolutional layer;

[0057] The hierarchical feature fusion unit and the fusion feature upsampling unit are cascaded to form the high-resolution image reconstruction module , its function is expressed as follows:

[0058] .

[0059] The shallow feature extraction module, the light field empty angle feature reconstruction module, the light field structure feature reconstruction module and the high-resolution image reconstruction module are cascaded to form a light field image super-resolution reconstruction network, which is used to reconstruct the input low-resolution light field image to be reconstructed into a high-resolution light field image, and its calculation process is as follows:

[0060] ;

[0061] in, represents the low-resolution light field image to be reconstructed, represents the reconstructed high-resolution light field image, , Indicates the size of the angle dimension, , Indicates the size of the space dimension, express or The scale factor, The shallow feature extraction module, represents the light field empty angle feature reconstruction module, represents the light field structure feature reconstruction module, represents the high-resolution image reconstruction module.

[0062] A light field image super-resolution reconstruction device based on a state model, comprising:

[0063] A shallow feature extraction module, used for extracting shallow features from a sub-aperture image of an input light field image to be reconstructed;

[0064] The deep feature extraction unit is built using the state-space model;

[0065] A light field empty angle feature reconstruction module is constructed based on the deep feature extraction unit and is used to reconstruct the light field empty angle feature based on the shallow feature;

[0066] A light field structure feature reconstruction module, constructed based on the deep feature extraction unit, and used to reconstruct light field structure features for the shallow features;

[0067] The high-resolution image reconstruction module is used to perform hierarchical feature fusion and up-sampling on the shallow features, the light field empty angle features and the light field structure features to obtain a reconstructed high-resolution light field image.

[0068] It can be seen from the above description of the present invention that, compared with the prior art, the present invention has the following beneficial effects:

[0069] 1. The present invention uses a light field empty angle feature reconstruction module to simultaneously model the interdependence between features from the spatial and angular dimensions of the light field image, thereby achieving full interaction of empty angle information.

[0070] 2. The present invention effectively explores the intrinsic structural characteristics of light field images and enhances the global light field structural feature expression capability of the LFSR model through a light field structural feature reconstruction module.

[0071] 3. The present invention uses a high-resolution image reconstruction module to mine complementary multi-scale features from shallow features, light field spatial angle features, and light field structure features, thereby improving the detail reconstruction capability of the LFSR model.

[0072] 4. The present invention maps 4D light field images into three 2D image representations: sub-aperture images (SAI), macro-pixel images (MacPI), and epipolar plane images (EPI). The state-space model is used to fully interact with the spatial information and angular information in the light field image, effectively exploring the intrinsic structural characteristics of the light field image, thereby greatly improving the reconstruction quality of low-resolution light field images. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a flow chart of the method of the present invention.

[0074] Figure 2 It is a schematic diagram of the shallow feature extraction module provided by the present invention.

[0075] FIG. 3( a ) is a schematic diagram of a selective state space module of the present invention.

[0076] FIG3( b ) is a schematic diagram of a deep feature extraction unit of the present invention.

[0077] FIG4( a ) is a schematic diagram of a spatial feature extraction unit of the present invention;

[0078] FIG4( b ) is a schematic diagram of an angle feature extraction unit of the present invention;

[0079] FIG4( c ) is a schematic diagram of a light field empty angle feature reconstruction module provided by the present invention.

[0080] FIG5 (a) is a horizontal light field structure feature extraction unit provided by the present invention;

[0081] FIG5( b ) is a vertical light field structure feature extraction unit provided by the present invention;

[0082] FIG5( c ) is a schematic diagram of a light field structure feature reconstruction module provided by the present invention.

[0083] Figure 6 It is a schematic diagram of a high-resolution image reconstruction module provided by the present invention.

[0084] Figure 7 It is a schematic diagram of a light field image super-resolution reconstruction network provided by the present invention.

[0085] Figure 8 The figure compares the reconstruction results of the central perspective of the Red_&_White_Building scene using the method provided by the present invention and other methods.

[0086] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0087] The present invention is further described below through specific implementation modes.

[0088] See also Figure 1 , a light field image super-resolution reconstruction method based on a state space model, comprising:

[0089] S10 constructs a shallow feature extraction module for extracting shallow features from the input light field image to be reconstructed in the form of a sub-aperture image.

[0090] See also Figure 2 In this step, the angle dimension is obtained as , And the spatial dimension is , The low-resolution light field image to be reconstructed , convert it into a sub-aperture image As input. Construct shallow feature extraction module Used to extract shallow features , which is expressed as follows:

[0091] ;

[0092] ;

[0093] in, represents the sub-aperture image form of the light field image to be reconstructed, Represents shallow features, represents the shallow feature extraction module, Indicates that the convolution kernel size is The convolutional layer does not change the number of channels of the input feature map. represents the convolutional layer function for channel upsampling, which upsamples the number of channels of the input feature map to 64. Represents the LReLU activation function.

[0094] S20 uses state-space model to build deep feature extraction unit.

[0095] In this step, referring to FIG3(a), the state space model includes parameterization and discretization, and the parameterization calculation process is as follows:

[0096]

[0097] in, , Respectively represent the length The input sequence and output sequence of Indicates the length is The learnable hidden state space of represents the hidden state space at the previous moment, , and represents the discretized parameter matrix, and The value is usually 64. Represents a real number.

[0098] Discretization uses the zero-order hold rule to discretize the continuous parameter matrix. The calculation process is as follows:

[0099]

[0100] in, represents the time step, , and Represents the discretized parameter matrix.

[0101] Integrating the above formulas can obtain the discretized state space model , the calculation process is as follows:

[0102]

[0103] in, Represents the hidden state space of the previous moment and the hidden state space of the current moment The hidden state space at the previous moment Calculated.

[0104] Referring to Figure 3(b), the deep feature extraction unit constructed using the state space model is as follows:

[0105] ;

[0106] ; ;

[0107] in, Represents shallow features of light field images, Represents the local intermediate features of the light field image, represents the depth features of the light field image, represents the deep feature extraction unit, The function representing the normalization of the layer, express The convolutional layer, The function representing the channel attention, , represents a learnable residual connection, represents the selective state-space model.

[0108] Furthermore, the selective state space model The expression is as follows:

[0109]

[0110] in, represents the weighted features of the branch output where the state space model is located, represents the supplementary features of the non-state-space model branch output, Indicates that the feature , The final refined features obtained by fusion, represents a linear mapping layer, represents depthwise separable convolution, represents the activation function, represents the state space model, represents the Hadamard dot product.

[0111] S30 constructs a light field empty angle feature reconstruction module based on the deep feature extraction unit, which is used to reconstruct the light field empty angle feature for the shallow feature.

[0112] Referring to FIG. 4 (a) to FIG. 4 (c), the steps specifically include the following:

[0113] S31 builds spatial feature extraction unit , which includes a matrix dimension conversion unit and deep feature extraction unit The cascade structure is as follows:

[0114] ;

[0115] in, Represents shallow features, Represents the matrix dimension transformation operation to transform shallow features Convert to sub-aperture image features , Represents the deep feature extraction unit.

[0116] S32 builds angle feature extraction unit , which includes a matrix dimension conversion unit and deep feature extraction unit The cascade structure is described as follows:

[0117] ;

[0118] in, Describe the sub-aperture image characteristics, Represents the matrix dimension transformation operation to transform sub-aperture image features into macro-pixel image features .

[0119] S33 integrates the spatial feature extraction unit and the angle feature extraction unit to reconstruct the light field empty angle feature. The light field empty angle feature reconstruction module is composed of the spatial feature extraction unit and angle feature extraction unit Alternate cascade three times and residually connect shallow features The composition is expressed as follows:

[0120] ;

[0121] ;

[0122] in, Represents the function of the light field empty angle feature reconstruction module, Indicates that shallow features are reconstructed by the light field empty angle feature reconstruction module The post-reconstruction light field empty angle features.

[0123] S40 constructs a light field structure feature reconstruction module based on the deep feature extraction unit, which is used to reconstruct the light field structure features of the shallow features; see Figure 5 (a)-Figure 5 (c), which specifically includes the following:

[0124] S41 builds horizontal light field structure feature extraction unit , which uses a matrix dimension transformation unit and deep feature extraction unit Cascaded, as shown below:

[0125] ;

[0126] in, represents the reconstructed light field empty angle feature, Represents the matrix dimension transformation operation to transform the light field empty angle feature Transformed into horizontal epipolar plane image features , Represents the deep feature extraction unit.

[0127] S42 builds vertical light field structure feature extraction unit , which uses a matrix dimension transformation unit and deep feature extraction unit The cascade structure is as follows:

[0128] ;

[0129] in, represents the matrix dimension transformation operation to transform the horizontal epipolar plane image features Transformed into vertical epipolar plane image features .

[0130] S43 extracts the horizontal light field structure feature and vertical light field structure feature extraction unit Alternate cascade three times and residually connect the reconstructed light field empty angle features The light field structure feature reconstruction module is constructed as follows:

[0131] ;

[0132] ;

[0133] in, represents the light field structure feature reconstruction module, Represents the light field empty angle feature reconstructed by go through Reconstructed light field structural features.

[0134] S50 builds a high-resolution image reconstruction module, which is used to perform hierarchical feature fusion and up-sampling of shallow features, light field spatial angle features and light field structural features to reconstruct a high-resolution light field image.

[0135] See also Figure 6 , this step specifically includes the following:

[0136] S51 constructs a hierarchical feature fusion unit. A function representing a hierarchical feature fusion unit, which transforms the input shallow features, reconstructed light field angle features, and reconstructed light field structure features into light field fusion features , which is expressed as follows:

[0137] ;

[0138] ;

[0139] in, Indicates that the convolution kernel size is The convolutional layer, represents the channel dimension concatenation operation, , , They represent shallow features, reconstructed light field empty angle features, and reconstructed light field structure features respectively.

[0140] S52 constructs a fusion feature upsampling unit. The function representing the fusion feature upsampling unit is expressed as follows:

[0141] ;

[0142] in, The scale factor is or Sub-pixel convolutional layer; Indicates that after the hierarchical feature fusion unit The obtained light field fusion features.

[0143] S53 cascaded hierarchical feature fusion units and fusion feature upsampling units constitute a high-resolution image reconstruction module , its function is expressed as follows:

[0144] .

[0145] In the present invention, a shallow feature extraction module, a light field empty angle feature reconstruction module, a light field structure feature reconstruction module and a high-resolution image reconstruction module are cascaded to form a light field image super-resolution reconstruction network, which is used to reconstruct the input low-resolution light field image to be reconstructed into a high-resolution light field image. The calculation process is as follows:

[0146] ;

[0147] in, represents the low-resolution light field image to be reconstructed, represents the reconstructed high-resolution light field image, , Indicates the size of the angle dimension, , Indicates the size of the space dimension, express or The scale factor, Shallow feature extraction module, represents the light field empty angle feature reconstruction module, represents the light field structure feature reconstruction module, Represents the high-resolution image reconstruction module.

[0148] Based on this, the present invention also proposes a state model-based light field image super-resolution reconstruction device, which is used to execute the above-mentioned state model-based light field image super-resolution reconstruction method, see Figure 7 ,include:

[0149] The shallow feature extraction module is used to extract shallow features from the input sub-aperture image of the light field image to be reconstructed. The shallow feature extraction module is used to execute step S10 of the above method.

[0150] The light field empty angle feature reconstruction module is constructed based on the deep feature extraction unit and is used to reconstruct the light field empty angle feature from the shallow feature. The light field empty angle feature reconstruction module is used to perform step S20 of the above method.

[0151] The light field structure feature reconstruction module is constructed based on the deep feature extraction unit and is used to reconstruct the light field structure feature of the shallow feature. The light field structure feature reconstruction module is used to perform step S40 of the above method.

[0152] The high-resolution image reconstruction module is used to perform hierarchical feature fusion and up-sampling on the shallow features, light field empty angle features and light field structure features to obtain a reconstructed high-resolution light field image. The high-resolution image reconstruction module is used to execute step S50 of the above method.

[0153] In order to better illustrate the advantages of the technical solution of the present invention, the following experiments are disclosed in this embodiment.

[0154] In order to achieve a comprehensive comparison of quantitative and qualitative performance, this experiment trained and verified the proposed method on 5 public light field datasets, which contain 144 training scenes and 23 test scenes, covering different light field contents and disparity distributions. Specifically, EPFL, HCInew, HCIold, INRIA and STFgantry were used to construct the benchmark datasets. The specific configuration of the training set and test set division of the benchmark dataset is shown in Table 1.

[0155] Table 1 Training dataset and test dataset

[0156]

[0157] In the experiment, Bicubic was used as a benchmark and compared with the classic single image super-resolution method RCAN based on deep learning and the most advanced light field image super-resolution reconstruction methods DistgSSR, LFT and EPIT. Among them, DistgSSR is a CNN-based light field image super-resolution method, and LFT and EPIT are Transformer-based light field image super-resolution methods. The corresponding experimental data are obtained based on the code and model training provided by the authors of the existing papers. On the test dataset The super-resolution results are shown in Table 2. The data in the table are the average PSNR and SSIM of all test light field images in the corresponding dataset. The bold data are the optimal data, and the underlined data are the suboptimal data.

[0158] Experimental results show that the proposed method performs best on five data sets, and its average PSNR and SSIM scores are 0.26dB and 0.19% higher than those of the most advanced light field image super-resolution reconstruction method EPIT.

[0159] Table 2 Different algorithms Average PSNR(dB) / SSIM comparison of super-resolution results

[0160]

[0161] In addition, the present invention provides Qualitative results obtained by different methods on super-resolution tasks. Figure 8 As shown, the reconstruction results of the scene Red_&_White_Building by the method of the present invention (i.e., Ours in the figure) and the existing advanced light field image super-resolution reconstruction methods are shown. Experimental results show that the method proposed in the present invention is better at restoring sharp edges and complex textures, which is not possible for most advanced light field image super-resolution reconstruction methods. For example, in the Red_&_White_Building scene from the EPFL dataset, it can be observed that only the method proposed in the present invention can reconstruct a clear and complete red diagonal line of the no parking sign. The reason is that the present invention maps the 4D light field image into three 2D image representations of sub-aperture, macro-pixel, and epipolar plane, and uses the state-space model to fully interact with the spatial information and angular information in the light field image, effectively exploring the intrinsic structural characteristics of the light field image, thereby greatly improving the reconstruction quality of low-resolution light field images.

[0162] In order to further verify the computational efficiency of the method of the present invention, Table 3 shows the comparison of the Params parameters and FLOPs calculation amount used by various methods. It can be seen that the method of the present invention consumes about 7 times less parameters than RCAN on the basis of ensuring the quality of light field super-resolution reconstruction image, and is also significantly lower than the DistgSSR method. The calculation amount is about 6 times less than RCAN, which is equivalent to DistgSSR.

[0163] Table 3 Params and FLOPs of different algorithms

[0164]

[0165] In summary, through experimental testing and analysis of real light field images and synthetic light field images of 144 training scenes and 23 test scenes, this experiment shows that the method of the present invention has great advantages in both subjective vision and objective performance. The super-resolution reconstruction of light field images is of high quality, and the edge and complex texture details are well restored. In addition, higher PSNR and SSIM performance are achieved with lower computational cost, which is better than the current most advanced light field image super-resolution reconstruction method.

[0166] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0167] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0168] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0169] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A light field image super-resolution reconstruction method based on a state model, characterized in that: include: Constructing a shallow feature extraction module for extracting shallow features from a sub-aperture image of an input light field image to be reconstructed; Using state-space models to build deep feature extraction units; Constructing a light field empty angle feature reconstruction module based on the deep feature extraction unit, for reconstructing the light field empty angle feature for the shallow feature; Constructing a light field structure feature reconstruction module based on the deep feature extraction unit, for reconstructing light field structure features for the shallow features; A high-resolution image reconstruction module is constructed to perform hierarchical feature fusion and up-sampling on the shallow features, the light field empty angle features and the light field structure features to obtain a reconstructed high-resolution light field image.

2. The state model-based light field image super-resolution reconstruction method according to claim 1, characterized in that: The shallow feature extraction module constructed is expressed as follows: ； in, represents the sub-aperture image form of the light field image to be reconstructed, Represents shallow features, represents the shallow feature extraction module, Indicates that the convolution kernel size is The convolutional layer, represents the convolutional layer function for channel upsampling, Represents the LReLU activation function.

3. The state model-based light field image super-resolution reconstruction method according to claim 1, characterized in that: The deep feature extraction unit constructed using the state space model is as follows: ； ； ； in, Represents shallow features of light field images, Represents the local intermediate features of the light field image, represents the depth features of the light field image, represents the deep feature extraction unit, The function representing the normalization of the layer, express The convolutional layer, The function representing the channel attention, , represents a learnable residual connection, represents the selective state-space model.

4. The state model-based light field image super-resolution reconstruction method according to claim 3, characterized in that: The Selectivity State Space Model The expression is as follows: ； Wherein, the selective state space model The input is the input feature of the layer standardization , the output is the final refined feature; represents a linear mapping layer, represents depthwise separable convolution, represents the activation function, represents the state space model, represents the Hadamard dot product.

5. The state model-based light field image super-resolution reconstruction method according to claim 4, characterized in that: The state space model To use the operator The processed sequence As input sequence, and mapped to output sequence , , the calculation process is as follows: ； in, , Respectively represent the length The input sequence and output sequence of Indicates the length The learnable hidden state space of represents the hidden state space at the previous moment, , and represents the discretized parameter matrix, Represents a real number.

6. The state model-based light field image super-resolution reconstruction method according to claim 1, characterized in that: A light field empty angle feature reconstruction module is constructed based on the deep feature extraction unit, specifically including the following: Constructing spatial feature extraction unit , which includes a matrix dimension conversion unit and deep feature extraction unit The cascade structure is as follows: ； Among them, the spatial feature extraction unit The input is the shallow feature , represents a matrix dimension transformation operation to transform the shallow features into sub-aperture image features, represents the deep feature extraction unit; Constructing the angle feature extraction unit , which includes a matrix dimension conversion unit and deep feature extraction unit The cascade structure is described as follows: ； Among them, the angle feature extraction unit The input is the sub-aperture image feature , representing a matrix dimension conversion operation to convert the sub-aperture image features into macro-pixel image features; The spatial feature extraction unit and the angle feature extraction unit are integrated to reconstruct the light field spatial angle feature, which is expressed as follows: ； ； in, Represents the function of the light field empty angle feature reconstruction module, Represents the reconstructed light field empty angle feature.

7. The state model-based light field image super-resolution reconstruction method according to claim 1, characterized in that: A light field structure feature reconstruction module is constructed based on the deep feature extraction unit, which specifically includes the following: Constructing horizontal light field structure feature extraction unit , which uses a matrix dimension transformation unit and deep feature extraction unit Cascaded, as shown below: ； Among them, the horizontal light field structure feature extraction unit The input is the reconstructed light field empty angle feature , Represents the matrix dimension transformation operation to transform the light field empty angle feature Transformed into horizontal epipolar plane image features , represents the deep feature extraction unit; Constructing a vertical light field structure feature extraction unit , which uses a matrix dimension transformation unit and deep feature extraction unit The cascade structure is as follows: ； Among them, the vertical light field structure feature extraction unit The input is the horizontal epipolar plane image feature , represents the matrix dimension transformation operation to transform the horizontal epipolar plane image features Transformed into vertical epipolar plane image features; The horizontal light field structure feature extraction unit and the vertical light field structure feature extraction unit Alternately cascade three times and reconstruct the light field empty angle feature by residual connection The light field structure feature reconstruction module is constructed as follows: ； ； in, represents the light field structure feature reconstruction module, Represents the reconstructed light field structure features.

8. The state model-based light field image super-resolution reconstruction method according to claim 1, characterized in that: Constructing a high-resolution image reconstruction module for hierarchical feature fusion and upsampling of the shallow features, the light field empty angle features, and the light field structure features to obtain a reconstructed high-resolution light field image, specifically including the following: Construct a hierarchical feature fusion unit, which converts the input shallow features, the reconstructed light field empty angle features, and the reconstructed light field structure features into light field fusion features , which is expressed as follows: ； ； in, Indicates that the convolution kernel size is The convolutional layer, represents the channel dimension concatenation operation, , , Respectively represent shallow features, reconstructed light field empty angle features, and reconstructed light field structure features; Construct a fusion feature upsampling unit, The function representing the fusion feature upsampling unit is expressed as follows: ； Among them, the function of the fusion feature upsampling unit The input is the light field fusion feature , The scale factor is or Sub-pixel convolutional layer; The hierarchical feature fusion unit and the fusion feature upsampling unit are cascaded to form the high-resolution image reconstruction module , its function is expressed as follows: 。 9. The state model-based light field image super-resolution reconstruction method according to claim 1, characterized in that: The shallow feature extraction module, the light field empty angle feature reconstruction module, the light field structure feature reconstruction module and the high-resolution image reconstruction module are cascaded to form a light field image super-resolution reconstruction network, which is used to reconstruct the input low-resolution light field image to be reconstructed into a high-resolution light field image, and its calculation process is as follows: ； in, represents the low-resolution light field image to be reconstructed, represents the reconstructed high-resolution light field image, , Indicates the size of the angle dimension, , Indicates the size of the space dimension, express or The scale factor, The shallow feature extraction module, represents the light field empty angle feature reconstruction module, represents the light field structure feature reconstruction module, represents the high-resolution image reconstruction module.

10. A light field image super-resolution reconstruction device based on a state model, characterized in that: include: A shallow feature extraction module, used for extracting shallow features from a sub-aperture image of an input light field image to be reconstructed; The deep feature extraction unit is built using the state-space model; A light field empty angle feature reconstruction module is constructed based on the deep feature extraction unit and is used to reconstruct the light field empty angle feature based on the shallow feature; A light field structure feature reconstruction module, constructed based on the deep feature extraction unit, and used to reconstruct light field structure features for the shallow features; The high-resolution image reconstruction module is used to perform hierarchical feature fusion and up-sampling on the shallow features, the light field empty angle features and the light field structure features to obtain a reconstructed high-resolution light field image.

Images