A light field image super-resolution method and system based on a channel separation state space model, a terminal and a storage medium

Abstract

The application discloses a kind of based on channel separation state space model's light field image super-resolution method, system, terminal and storage medium, the method includes: input original resolution light field image, and original resolution light field image is carried out initial feature extraction and depth feature extraction, obtain initial high-dimensional feature and target depth feature;Target depth feature is carried out structure feature extraction, and target structure feature is obtained;Initial high-dimensional feature, target depth feature and target structure feature are carried out feature splicing processing and feature fusion processing, and fusion feature is obtained;Fusion feature is carried out channel number adjustment processing and pixel shuffling processing, and target resolution light field image is obtained.The application can effectively improve the resolution of original resolution light field image by initial feature extraction, depth feature extraction, structure feature extraction and feature fusion processing to low resolution light field image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, terminal, and computer-readable storage medium for super-resolution of light field images based on a channel separation state space model. Background Technology

[0002] Light field images are an image format capable of recording information about three-dimensional scenes. They can simultaneously capture the intensity and direction of light rays at different locations in space, enabling the perception of scene depth and viewpoint transformation, thus possessing wide-ranging application value. However, due to the unique structure of light field cameras, the spatial resolution of light field images decreases while acquiring multi-viewpoint images, which limits their applications to some extent. Therefore, researching methods to increase the spatial resolution of light field images is of great significance.

[0003] To address the aforementioned issues, current technologies typically employ deep learning techniques and neural networks to learn the mapping from low-resolution light field images to high-resolution light field images. Most mainstream light field image super-resolution networks are based on the Transformer architecture.

[0004] However, the computational complexity of the self-attention mechanism in the Transformer architecture is quadratic with that of the input image, resulting in high computational complexity and making it difficult to improve the super-resolution of light field images.

[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0006] The main objective of this invention is to provide a method, system, terminal, and computer-readable storage medium for super-resolution of light field images based on a channel-separated state-space model. This invention aims to solve the problem that the computational complexity of the self-attention mechanism in the Transformer architecture is quadratically related to the input image, resulting in high computational complexity and making it difficult to improve the super-resolution of light field images.

[0007] To achieve the above objectives, this invention provides a method for super-resolution of light field images based on a channel-separated state-space model. The method includes the following steps: Input the original resolution light field image, and perform initial feature extraction and depth feature extraction on the original resolution light field image to obtain initial high-dimensional features and target depth features; Structural features are extracted from the depth features of the target to obtain the target structural features; The initial high-dimensional features, the target depth features, and the target structural features are subjected to feature concatenation and feature fusion processing to obtain fused features; The fused features are subjected to channel number adjustment and pixel shuffling to obtain a light field image with target resolution.

[0008] Optionally, the light field image super-resolution method based on the channel separation state space model, wherein the input is an original resolution light field image, and initial feature extraction and depth feature extraction are performed on the original resolution light field image to obtain initial high-dimensional features and target depth features, specifically includes: Input the original resolution light field image, and use an initial convolution and three cascaded convolutional networks to perform initial feature extraction on the original resolution light field image to obtain initial high-dimensional features; A channel-separated state space model is determined, and the initial high-dimensional features are input into the feature extraction module of the channel-separated state space model. The initial high-dimensional features are extracted by the feature extraction module to extract angular and spatial features, thereby obtaining target angular features and target spatial features, and target depth features are obtained based on the target angular features and target spatial features.

[0009] Optionally, the light field image super-resolution method based on the channel separation state space model, wherein the step of extracting angular and spatial features from the initial high-dimensional features through the feature extraction module to obtain target angular features and target spatial features, and obtaining target depth features based on the target angular features and target spatial features, specifically includes: The initial high-dimensional features are merged into the batch dimension to obtain a macro-pixel image, and the macro-pixel image is input into the feature extraction module; The feature extraction module models the relationships between pixels at different angles in the macro-pixel image to obtain angle information. The expression for the angle information is as follows: ; in, For angle information, For feature extraction module, Macro-pixel image; The angle dimension of the angle information is merged into the batch dimension to obtain a sub-aperture image, and the sub-aperture image is input into the feature extraction module; The feature extraction module models the relationships between pixels at different angles in the sub-aperture image to obtain spatial information. The expression for the spatial information is as follows: ; in, For spatial information, Sub-aperture image; The target depth features are obtained by iterative processing based on the angle information and the spatial information. The expression for the target depth feature is as follows: ; in, For target depth features, This is the sixth input feature extraction module. For the first input to the feature extraction module, These are the initial high-dimensional features.

[0010] Optionally, the light field image super-resolution method based on the channel separation state space model, wherein the step of extracting structural features from the target depth features to obtain target structural features specifically includes: The horizontal angle dimension and width dimension of the target depth feature are merged into the batch dimension to obtain a vertical polar plane image, and the vertical polar plane image is input into the feature extraction module; The feature extraction module extracts the first light field structure information from the vertical epiplane image and merges the vertical angle dimension and length dimension in the first light field structure information into the batch dimension to obtain the horizontal epiplane image. The expression for the first light field structure information is as follows: ; in, This is the information about the first optical field structure. This is a vertical polar plane image; The horizontal polar plane image is input to the feature extraction module, and the second light field structure information in the horizontal polar plane image is extracted by the feature extraction module. The expression for the second light field structure information is as follows: ; in, This is information about the structure of the second light field. A horizontal polar plane image; The target structural features are obtained by iterative processing based on the first and second optical field structure information. The expression for the target structural feature is as follows: ; in, The target structural features.

[0011] Optionally, in the light field image super-resolution method based on the channel separation state space model, the processing procedure of the feature extraction module for each input feature includes: For any light field feature input into the feature extraction module, the feature extraction module uses layer normalization to normalize the arbitrary light field feature to obtain normalized features; The normalized features are subjected to long-distance relationship modeling to obtain modeling features; The modeling features are added to the initial high-dimensional features to obtain the first added modeling features; The first additive modeling features are enhanced by layer normalization, a convolutional layer, and a channel attention module to extract local features and interact with channel information to obtain interactive features. The interaction features are added to the initial high-dimensional features to obtain the second added modeling features.

[0012] Optionally, the light field image super-resolution method based on the channel separation state space model, wherein the step of performing a first addition process on the modeling features and the initial high-dimensional features to obtain the first added modeling features specifically includes: The normalized features are processed using a linear layer and the SiLU activation function to obtain the first feature map; The normalized features are processed using a linear layer, a depthwise separable convolution, and the SiLU activation function to obtain the basic features; The basic features are extracted using a two-dimensional selective scanning mechanism and then input into a layer normalization process for feature enhancement to obtain a second feature map. The first feature map and the second feature map are multiplied element-wise and fused to obtain the first additive modeling feature.

[0013] Optionally, the light field image super-resolution method based on the channel separation state space model, wherein the channel number adjustment and pixel shuffling processing of the fused features to obtain the target resolution light field image specifically includes: A convolutional layer is used to adjust the number of channels in the fused feature to obtain the adjusted fused feature. The adjusted fusion features are subjected to light field image enlargement processing using a pixel shuffling operation to obtain enlarged fusion features, and the enlarged fusion features are subjected to channel number reduction processing using a convolutional network to obtain the target fusion features; The original resolution light field image is downsampled to obtain a downsampled light field image, and the downsampled light field image is added to the target fusion feature to obtain the target resolution light field image. The expression for the target resolution light field image is: ; in, For the target resolution light field image, For downsampling processing, This is the original resolution light field image. For convolutional networks, Pixel shuffling operation, Features are fused to target specific features.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a light field image super-resolution system based on a channel-separated state space model, wherein the light field image super-resolution system based on the channel-separated state space model includes: The target depth feature extraction module is used to input the original resolution light field image and perform initial feature extraction and depth feature extraction on the original resolution light field image to obtain initial high-dimensional features and target depth features; The structural feature extraction module is used to extract structural features from the target depth features to obtain the target structural features; The fusion feature generation module is used to perform feature concatenation and feature fusion processing on the initial high-dimensional features, the target depth features, and the target structural features to obtain fused features; The target resolution light field image generation module is used to perform channel number adjustment and pixel shuffling processing on the fused features to obtain the target resolution light field image.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a light field image super-resolution program based on a channel-separated state space model stored in the memory and executable on the processor. When the light field image super-resolution program based on a channel-separated state space model is executed by the processor, it implements the steps of the light field image super-resolution method based on a channel-separated state space model as described above.

[0016] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a light field image super-resolution program based on a channel-separated state space model, and when the light field image super-resolution program based on the channel-separated state space model is executed by a processor, it implements the steps of the light field image super-resolution method based on the channel-separated state space model as described above.

[0017] In this invention, an original resolution light field image is input, and initial feature extraction and depth feature extraction are performed on the original resolution light field image to obtain initial high-dimensional features and target depth features. Structural features are extracted from the target depth features to obtain target structural features. Feature stitching and feature fusion processing are performed on the initial high-dimensional features, the target depth features, and the target structural features to obtain fused features. Channel number adjustment and pixel shuffling processing are performed on the fused features to obtain the target resolution light field image. This invention effectively improves the resolution of the original resolution light field image by performing initial feature extraction, depth feature extraction, structural feature extraction, and feature fusion processing on a low-resolution light field image. Attached Figure Description

[0018] Figure 1 This is a flowchart of a preferred embodiment of the optical field image super-resolution method based on the channel separation state space model of the present invention; Figure 2 This is a schematic diagram of the framework of the optical field image super-resolution method based on the channel separation state space model of the present invention. Figure 3 This is a schematic diagram showing a qualitative visualization comparison between the present invention and existing super-resolution methods based on a preferred embodiment of the light field image super-resolution method of the present invention and a preferred embodiment of the present invention based on a channel separation state space model. Figure 4 This is a structural diagram of a preferred embodiment of the optical field image super-resolution system based on the channel separation state space model of the present invention; Figure 5 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0020] Light field images are an image format capable of recording information about three-dimensional scenes. They can simultaneously capture the intensity and direction of light rays at different locations in space, enabling the perception of scene depth and viewpoint transformation, thus possessing wide-ranging application value. However, due to the unique structure of light field cameras, acquiring multi-viewpoint images leads to a decrease in the spatial resolution of light field images, which limits their applications to some extent. Therefore, researching methods to increase the spatial resolution of light field images is of great significance.

[0021] Current mainstream methods for light field image super-resolution utilize deep learning techniques and neural networks to learn the mapping from low-resolution to high-resolution light field images. Early network models were primarily based on convolutional neural networks; however, these methods had limited effectiveness. Most current mainstream light field image super-resolution networks are based on the Transformer architecture to extract long-range dependencies in the light field. However, the self-attention mechanism in the Transformer architecture has a computational complexity that is quadratic with the input image, and this high computational cost limits its practical application.

[0022] To address the aforementioned problems, this invention proposes a light field image super-resolution method based on a channel-separated state-space model, belonging to the field of image super-resolution in image processing technology. The proposed light field image super-resolution method aims to increase the spatial resolution of each sub-aperture image of a 5×5 light field image by a factor of 2 and 4, respectively. This invention employs a deep learning-trained neural network approach, comprising four components: 1. Extracting basic feature information (i.e., the initial high-dimensional features) from the input light field image (i.e., the original resolution light field image in this invention) through an initial convolution and three cascaded convolutional networks. 2. Using the light field image feature extraction module based on the channel-separated state-space model proposed in this invention, alternatingly extracting spatial and angular features (i.e., the target depth features in this invention) from the sub-aperture images and macro-pixel images of the light field image. 3. Using the light field image feature extraction module based on the channel-separated state-space model proposed in this invention, alternatingly extracting structural features of the light field image (i.e., the target structural features in this invention) from the horizontal and vertical epipolar plane images. 4. A convolutional layer is used to fully fuse the light field features obtained in the first three steps, and the feature map (i.e., the original resolution light field image in this invention) is reconstructed through an upsampling module to obtain the final high-resolution light field image (i.e., the target resolution light field image in this invention).

[0023] The method of this invention has clear modules and simple steps, and can fully utilize the spatial, angular, and structural information of the light field image to reconstruct a clear, high-resolution light field image. Furthermore, the light field image feature extraction module based on the channel-separated state-space model of this invention can effectively reduce the computational complexity of the model and improve its computational efficiency.

[0024] The preferred embodiment of the light field image super-resolution method based on the channel separation state space model described in this invention, such as... Figure 1 and Figure 2 As shown, the light field image super-resolution method based on the channel separation state space model includes the following steps: Step S10: Input the original resolution light field image, and perform initial feature extraction and depth feature extraction on the original resolution light field image to obtain initial high-dimensional features and target depth features.

[0025] Specifically, the original resolution light field image is input, and an initial convolutional network and three cascaded convolutional networks are used to extract initial features from the original resolution light field image to obtain initial high-dimensional features.

[0026] A channel-separated state-space model is determined, and the initial high-dimensional features are input into the feature extraction module of the channel-separated state-space model; the initial high-dimensional features are merged into the batch dimension to obtain a macro-pixel image, and the macro-pixel image is input into the feature extraction module; the feature extraction module models the relationship between pixels at different angles in the macro-pixel image to obtain angle information; wherein, the expression of the angle information is: ;in, For angle information, For feature extraction module, The image is a macro-pixel image; the angular dimension of the angular information is merged into the batch dimension to obtain a sub-aperture image, and the sub-aperture image is input into the feature extraction module; the feature extraction module models the relationship between pixels at different angles in the sub-aperture image to obtain spatial information; wherein, the expression of the spatial information is: ;in, For spatial information, The image is a sub-aperture image; iterative processing is performed based on the angle information and the spatial information to obtain the target depth feature; wherein, the expression of the target depth feature is: ;in, For target depth features, This is the sixth input feature extraction module. For the first input to the feature extraction module, These are the initial high-dimensional features.

[0027] like Figure 2 As shown, firstly, this invention uses an initial convolutional network with a kernel size of 1×3×3 and three cascaded 1×3×3 convolutional networks to process an input image with a spatial resolution of... Angular resolution is Light field image (i.e., the original resolution light field image in this invention) is used for feature extraction to obtain initial high-dimensional features. ,in, For feature dimensions.

[0028] like Figure 2 As shown, the initial high-dimensional features As input, six feature extraction modules based on the channel separation state space model are used sequentially. The target depth features are obtained by alternately extracting angular and spatial features three times from the macro-pixel image and sub-aperture image representations of the light field image. .

[0029] Step S20: Extract structural features from the target depth features to obtain target structural features.

[0030] like Figure 2 As shown, based on the obtained target depth features, six feature extraction modules based on the channel separation state space model are used sequentially. The structural features of the light field are extracted three times alternately from the vertical epipolar plane image and the horizontal epipolar plane image representations of the light field image to obtain the target structural features, namely... .

[0031] Specifically, the horizontal angle dimension and width dimension of the target depth feature are merged into the batch dimension to obtain a vertical epipolar image, and the vertical epipolar image is input into the feature extraction module; the feature extraction module extracts the first light field structure information from the vertical epipolar image, and merges the vertical angle dimension and length dimension of the first light field structure information into the batch dimension to obtain a horizontal epipolar image; wherein, the expression of the first light field structure information is: ;in, This is the information about the first optical field structure. The image is a vertical epipolar plane image; the horizontal epipolar plane image is input to the feature extraction module, and the feature extraction module extracts the second light field structure information from the horizontal epipolar plane image; wherein, the expression of the second light field structure information is: ;in, This is information about the structure of the second light field. The horizontal epipolar image is iteratively processed based on the first and second light field structure information to obtain target structure features; wherein, the expression of the target structure features is: ;in, The target structural features.

[0032] This invention uses initial high-dimensional features Spatial dimension ( The data is merged into the batch dimension, thereby converting it into a macro-pixel representation (i.e., the macro-pixel image in this invention). .

[0033] Furthermore, the macro-pixel features (i.e., the macro-pixel image in this invention) are fed into the channel separation state space model feature extraction module of this invention to model the relationship between pixels at different angles in the macro-pixel image, thereby extracting angle information, i.e. .

[0034] Furthermore, by using the angle information obtained in the previous step Angle and dimension ( The data is merged into the batch dimension, thereby converting it into a sub-aperture image representation (i.e., the sub-aperture image in this invention). .

[0035] Furthermore, the sub-aperture features from the previous step (i.e., the sub-aperture image in this invention) are fed into the channel separation state space model feature extraction module of this invention. The feature extraction module models the relationships between pixels at different locations in the sub-aperture image, thereby extracting spatial information. .

[0036] Repeat the above steps three times to obtain the target depth features.

[0037] Furthermore, the present invention utilizes target depth features Horizontal angle dimension ( ) and wide dimension ( The data is merged into the batch dimension, thereby converting it into a vertical polar plane image representation (i.e., the vertical polar plane image in this invention). .

[0038] Furthermore, the vertical polar plane feature obtained in the previous step (i.e., the vertical polar plane image in this invention) is fed into the channel separation state space model feature extraction module in this invention. The feature extraction module extracts the light field structure information (i.e., the first light field structure information in this invention). .

[0039] Furthermore, the feature map obtained in the previous step... vertical angle dimension ( ) and long dimension ( The data is merged into the batch dimension, thereby converting it into a horizontal polar plane image representation (i.e., the horizontal polar plane image in this invention). .

[0040] Furthermore, the horizontal polar plane features obtained in the previous step (i.e., the horizontal polar plane image in this invention) are fed into the channel separation state space model feature extraction module of this invention. The feature extraction module extracts the light field structure information to obtain the second light field structure information, i.e. .

[0041] Repeat the above steps three times to obtain the target structural features.

[0042] Further, for any light field feature input to the feature extraction module, the feature extraction module performs layer normalization to normalize the arbitrary light field feature, obtaining normalized features; performs long-distance relationship modeling on the normalized features, obtaining modeled features; processes the normalized features using a linear layer and the SiLU activation function, obtaining a first feature map; processes the normalized features using a linear layer, a depthwise separable convolution, and the SiLU activation function, obtaining basic features; extracts features from the basic features using a two-dimensional selective scanning mechanism, and inputs them to the layer normalization module for feature enhancement, obtaining a second feature map; performs element-wise multiplication and feature fusion on the first feature map and the second feature map, obtaining a first additive modeling feature; performs enhanced local feature extraction and channel information interaction on the first additive modeling feature using layer normalization, a convolutional layer, and a channel attention module, obtaining interactive features; and performs a second additive process on the interactive features and the initial high-dimensional features, obtaining a second additive modeling feature.

[0043] The process of processing the input light field features by the feature extraction module based on the channel separation state space model in this invention includes the following steps: 1. Light field features for any representation of the input (In this invention, these can be initial high-dimensional features, target depth features, and target structural features.) Layer normalization (LN) is used to normalize the channel dimensions of the features, resulting in normalized features, i.e. ,in , Features Mean along the channel dimension, Features Variance along the channel dimension.

[0044] 2. Based on the obtained normalized features The Channel-Split State-Space Model (CSSSM) of this invention is used to normalize the features. Long-distance relationship modeling is performed to obtain modeling features. These modeling features are then added to the initial features (i.e., the initial high-dimensional features in this invention) to obtain the first added modeling features. ,in, It is a learnable scaling factor.

[0045] 3. Based on the obtained first summation modeling features Furthermore, layer normalization, a convolutional layer, and channel attention (CA) are used to enhance local feature extraction and channel information interaction. The resulting features (i.e., interaction features) are then added to the initial features (i.e., the initial high-dimensional features in this invention). ,in, It is a learnable scaling factor.

[0046] It is understood that the specific generation process of the first additive modeling feature in this invention is as follows: 1. For the input normalized features First, a feature map (i.e., the first feature map in this invention) is obtained using a linear layer and the SiLU activation function. .

[0047] 2. Normalized features of the input The normalized features are processed sequentially using a linear layer, a depthwise separable convolution (DWConv), and a SiLU activation function. These features are then subjected to a channel-separated two-dimensional selected scan (SS2D) mechanism for comprehensive feature extraction. Finally, an enhanced feature map (i.e., the second feature map in this invention) is obtained through a layer of normalization. The above process is as follows: .

[0048] 3. Based on the two obtained feature maps (i.e., the first feature map and the second feature map), perform element-wise multiplication and use a linear layer to obtain the fused feature (i.e., the first additive modeling feature). .

[0049] The specific processing steps for extracting normalized features through a two-dimensional selective scanning mechanism (SS2D) using channel separation are as follows: 1. For an input feature (This could refer to normalized features), firstly, its channel dimensions ( The data is divided into four equal parts, resulting in four feature maps. .

[0050] 2. Based on the obtained feature map The feature map is flattened into a pixel sequence in the following order: from top left to bottom right, from bottom right to top left, from bottom left to top right, and from top right to bottom left. .

[0051] 3. For the obtained pixel sequence We model it using a state-space model; specifically, the output... It can be obtained through the following equation: , .in, For state, Here is the state transition matrix. and This is the projection matrix.

[0052] Step S30: Perform feature splicing and feature fusion processing on the initial high-dimensional features, the target depth features, and the target structural features to obtain fused features.

[0053] Specifically, the present invention is based on the obtained multi-level features (i.e., the initial high-dimensional features). The target depth features and the target structural features First, the three features are concatenated by channels, and then a convolutional layer is used to fuse the features to obtain the fused features. .

[0054] Step S40: Perform channel number adjustment and pixel shuffling on the fused features to obtain a light field image with target resolution.

[0055] Specifically, a convolutional layer is used to adjust the number of channels in the fusion feature to obtain an adjusted fusion feature; a pixel shuffling operation is used to enlarge the light field image of the adjusted fusion feature to obtain an enlarged fusion feature; a convolutional network is then used to reduce the number of channels in the enlarged fusion feature to obtain the target fusion feature; the original resolution light field image is downsampled to obtain a downsampled light field image; and the downsampled light field image is then added to the target fusion feature to obtain the target resolution light field image; wherein, the expression for the target resolution light field image is: ;in, For the target resolution light field image, For downsampling processing, This is the original resolution light field image. For convolutional networks, Pixel shuffling operation, Features are fused to target specific features.

[0056] For the obtained fusion features This invention first uses a convolutional layer to adjust the fusion features. The number of channels is first increased, then a pixel shuffle operation is used to enlarge the resolution of the light field image, and then a convolutional network is used to reduce the number of channels to 1. Finally, it is added to the upsampled initial input light field image (i.e., the original resolution light field image in this invention) to obtain the high-resolution light field image (i.e., the target resolution light field image in this invention). .

[0057] Specific examples are as follows: 1. For experimental data, this invention uses five open-source light field image datasets: EPFL, HCInew, HCIold, INRIA, and STFgantry. EPFL and INRIA are real datasets captured by a Lytro all-light camera; STFgantry is a real-scene dataset captured by a gantry camera; and HCIold and HCInew are virtual light field datasets synthesized using simulation software. This invention uses bicubic interpolation to downsample high-resolution images by a factor of 2 and 4, respectively, as input to the neural network. The five datasets are divided into a training set of 144 scenes and a test set of 23 scenes.

[0058] 2. Regarding the evaluation method, this invention uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Measure (SSIM) as super-resolution evaluation metrics, as detailed below: ; ; in, The maximum possible pixel value in the image. The variance of pixel values. For image The brightness is defined as the average grayscale value; For image The contrast ratio is defined as the standard deviation of the grayscale values.

[0059] 3. Regarding the super-resolution results, the results of this invention on five datasets for 2x and 4x light field super-resolution tasks are shown in Table 1: Table 1: Results of this invention on five datasets for 2x and 4x light field super-resolution tasks.

[0060] like Figure 3 As shown, in addition to quantitative evaluation, this invention also performs a qualitative visual comparison with existing super-resolution methods, and the results are as follows. Figure 3 As shown.

[0061] Furthermore, such as Figure 4 As shown, based on the above-mentioned light field image super-resolution method based on the channel-separated state space model, the present invention also provides a light field image super-resolution system based on the channel-separated state space model, wherein the light field image super-resolution system based on the channel-separated state space model includes: The target depth feature extraction module 51 is used to input the original resolution light field image and perform initial feature extraction and depth feature extraction on the original resolution light field image to obtain initial high-dimensional features and target depth features; The structural feature extraction module 52 is used to extract structural features from the target depth features to obtain target structural features; The fusion feature generation module 53 is used to perform feature splicing and feature fusion processing on the initial high-dimensional features, the target depth features, and the target structural features to obtain fusion features; The target resolution light field image generation module 54 is used to perform channel number adjustment processing and pixel shuffling processing on the fused features to obtain the target resolution light field image.

[0062] Furthermore, such as Figure 5 As shown, based on the above-mentioned light field image super-resolution method and system based on the channel separation state space model, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0063] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a light field image super-resolution program 40 based on a channel-separated state space model. This light field image super-resolution program 40 based on a channel-separated state space model can be executed by the processor 10 to implement the light field image super-resolution method based on a channel-separated state space model in this application.

[0064] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the light field image super-resolution method based on the channel separation state space model.

[0065] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface.

[0066] In one embodiment, when the processor 10 executes the light field image super-resolution program 40 based on the channel-separated state space model in the memory 20, it implements the steps of the light field image super-resolution method based on the channel-separated state space model as described above.

[0067] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a light field image super-resolution program based on a channel-separated state space model, and the light field image super-resolution program based on the channel-separated state space model, when executed by a processor, implements the steps of the light field image super-resolution method based on the channel-separated state space model as described above.

[0068] In summary, this invention provides a method, system, terminal, and storage medium for super-resolution of light field images based on a channel-separated state-space model. The method includes: inputting an original resolution light field image; performing initial feature extraction and depth feature extraction on the original resolution light field image to obtain initial high-dimensional features and target depth features; extracting structural features from the target depth features to obtain target structural features; performing feature concatenation and feature fusion processing on the initial high-dimensional features, the target depth features, and the target structural features to obtain fused features; and performing channel number adjustment and pixel shuffling processing on the fused features to obtain the target resolution light field image. This invention effectively improves the resolution of the original resolution light field image by performing initial feature extraction, depth feature extraction, structural feature extraction, and feature fusion processing on the low-resolution light field image.

[0069] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0070] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0071] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for super-resolution of light field images based on a channel-separated state-space model, characterized in that, The light field image super-resolution method based on the channel separation state space model includes: Input the original resolution light field image, and perform initial feature extraction and depth feature extraction on the original resolution light field image to obtain initial high-dimensional features and target depth features; Structural features are extracted from the depth features of the target to obtain the target structural features; The initial high-dimensional features, the target depth features, and the target structural features are subjected to feature concatenation and feature fusion processing to obtain fused features; The fused features are subjected to channel number adjustment and pixel shuffling to obtain a light field image with target resolution.

2. The light field image super-resolution method based on a channel-separated state-space model according to claim 1, characterized in that, The input is a raw resolution light field image, and initial feature extraction and depth feature extraction are performed on the raw resolution light field image to obtain initial high-dimensional features and target depth features, specifically including: Input the original resolution light field image, and use an initial convolution and three cascaded convolutional networks to perform initial feature extraction on the original resolution light field image to obtain initial high-dimensional features; A channel-separated state space model is determined, and the initial high-dimensional features are input into the feature extraction module of the channel-separated state space model. The initial high-dimensional features are extracted by the feature extraction module to extract angular and spatial features, thereby obtaining target angular features and target spatial features, and target depth features are obtained based on the target angular features and target spatial features.

3. The light field image super-resolution method based on the channel separation state-space model according to claim 2, characterized in that, The step of extracting angular and spatial features from the initial high-dimensional features using the feature extraction module to obtain target angular features and target spatial features, and then obtaining target depth features based on the target angular and spatial features, specifically includes: The initial high-dimensional features are merged into the batch dimension to obtain a macro-pixel image, and the macro-pixel image is input into the feature extraction module; The feature extraction module models the relationships between pixels at different angles in the macro-pixel image to obtain angle information. The expression for the angle information is as follows: ； in, For angle information, For feature extraction module, Macro-pixel image; The angle dimension of the angle information is merged into the batch dimension to obtain a sub-aperture image, and the sub-aperture image is input into the feature extraction module; The feature extraction module models the relationships between pixels at different angles in the sub-aperture image to obtain spatial information. The expression for the spatial information is as follows: ； in, For spatial information, Sub-aperture image; The target depth features are obtained by iterative processing based on the angle information and the spatial information. The expression for the target depth feature is as follows: ； in, For target depth features, This is the sixth input feature extraction module. For the first input to the feature extraction module, These are the initial high-dimensional features.

4. The light field image super-resolution method based on the channel separation state space model according to claim 3, characterized in that, The step of extracting structural features from the target depth features to obtain target structural features specifically includes: The horizontal angle dimension and width dimension of the target depth feature are merged into the batch dimension to obtain a vertical polar plane image, and the vertical polar plane image is input into the feature extraction module; The feature extraction module extracts the first light field structure information from the vertical epiplane image and merges the vertical angle dimension and length dimension in the first light field structure information into the batch dimension to obtain the horizontal epiplane image. The expression for the first light field structure information is as follows: ； in, This is the information about the first optical field structure. This is a vertical polar plane image; The horizontal polar plane image is input to the feature extraction module, and the second light field structure information in the horizontal polar plane image is extracted by the feature extraction module. The expression for the second light field structure information is as follows: ； in, This is information about the structure of the second light field. A horizontal polar plane image; The target structural features are obtained by iterative processing based on the first and second optical field structure information. The expression for the target structural feature is as follows: ； in, The target structural features.

5. The light field image super-resolution method based on the channel separation state-space model according to claim 4, characterized in that, The processing procedure for each input feature by the feature extraction module includes: For any light field feature input into the feature extraction module, the feature extraction module uses layer normalization to normalize the arbitrary light field feature to obtain normalized features; The normalized features are subjected to long-distance relationship modeling to obtain modeling features; The modeling features are added to the initial high-dimensional features to obtain the first added modeling features; The first additive modeling features are enhanced by layer normalization, a convolutional layer, and a channel attention module to extract local features and interact with channel information to obtain interactive features. The interaction features are added to the initial high-dimensional features to obtain the second added modeling features.

6. The light field image super-resolution method based on the channel separation state-space model according to claim 5, characterized in that, The step of performing a first addition process between the modeling features and the initial high-dimensional features to obtain the first added modeling features specifically includes: The normalized features are processed using a linear layer and the SiLU activation function to obtain the first feature map; The normalized features are processed using a linear layer, a depthwise separable convolution, and the SiLU activation function to obtain the basic features; The basic features are extracted using a two-dimensional selective scanning mechanism and then input into a layer normalization process for feature enhancement to obtain a second feature map. The first feature map and the second feature map are multiplied element-wise and fused to obtain the first additive modeling feature.

7. The light field image super-resolution method based on a channel-separated state-space model according to claim 1, characterized in that, The process of adjusting the number of channels and shuffling pixels in the fused features to obtain the target resolution light field image specifically includes: A convolutional layer is used to adjust the number of channels in the fused feature to obtain the adjusted fused feature. The adjusted fusion features are subjected to light field image enlargement processing using a pixel shuffling operation to obtain enlarged fusion features, and the enlarged fusion features are subjected to channel number reduction processing using a convolutional network to obtain the target fusion features; The original resolution light field image is downsampled to obtain a downsampled light field image, and the downsampled light field image is added to the target fusion feature to obtain the target resolution light field image. The expression for the target resolution light field image is: ； in, For the target resolution light field image, For downsampling processing, This is the original resolution light field image. For convolutional networks, Pixel shuffling operation, Features are fused to target specific features.

8. A light field image super-resolution system based on a channel-separated state-space model, characterized in that, The light field image super-resolution system based on the channel separation state space model includes: The target depth feature extraction module is used to input the original resolution light field image and perform initial feature extraction and depth feature extraction on the original resolution light field image to obtain initial high-dimensional features and target depth features; The structural feature extraction module is used to extract structural features from the target depth features to obtain the target structural features; The fusion feature generation module is used to perform feature concatenation and feature fusion processing on the initial high-dimensional features, the target depth features, and the target structural features to obtain fused features; The target resolution light field image generation module is used to perform channel number adjustment and pixel shuffling processing on the fused features to obtain the target resolution light field image.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a light field image super-resolution program based on a channel-separated state space model stored in the memory and executable on the processor. When the light field image super-resolution program based on a channel-separated state space model is executed by the processor, it implements the steps of the light field image super-resolution method based on a channel-separated state space model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a light field image super-resolution program based on a channel-separated state space model, which, when executed by a processor, implements the steps of the light field image super-resolution method based on a channel-separated state space model as described in any one of claims 1-7.

Images