Method for spatial super-resolution of light field images based on multi-feature fusion

Abstract

The application relates to the technical field of image resolution technology and discloses a light field image space super-resolution method based on multi-feature fusion, which comprises the following steps: acquiring a light field image containing a plurality of sub-aperture images and converting the light field image into a macro-pixel image; performing spatial feature extraction on the sub-aperture images and performing angle feature extraction and limit feature extraction on the macro-pixel image to obtain spatial feature information, angle feature information and limit feature information; fusing and stacking the spatial feature information, the angle feature information and the limit feature information to obtain output features; and performing spatial up-sampling on the output features to obtain super-resolved sub-aperture images; and the method effectively improves the reconstruction resolution of the light field image.

Description

Technical Field

[0001] This invention relates to the technical field of image resolution technology, and specifically to a spatial super-resolution method for light field images based on multi-feature fusion. Background Technology

[0002] Light field cameras contain microlens arrays, which can record light passing through them from different directions. This allows light field cameras to capture light field images and, during the refocusing process in post-processing, obtain many images from different angles and positions while capturing a single light field image. Therefore, they are widely used in fields such as depth estimation, 3D reconstruction, object segmentation, and image refocusing.

[0003] Due to the spatial limitations of microlens arrays, the spatial resolution of sub-aperture images in light field images is much lower than that of two-dimensional images acquired by traditional imaging devices. Therefore, improving the spatial resolution of light field images has become a research hotspot in the field of image resolution technology.

[0004] In related technologies, stacked sub-aperture images or epipolar images (EPI) at different angles are often used to reconstruct high-resolution light field images. However, due to the complex four-dimensional nature of light field images, the above two methods cannot fully utilize the inherent angular and spatial information in the data, resulting in unsatisfactory reconstruction resolution. Summary of the Invention

[0005] To improve the reconstruction resolution of light field images, this invention provides a spatial super-resolution method for light field images based on multi-feature fusion.

[0006] In a first aspect, the present invention provides a spatial super-resolution method for light field images based on multi-feature fusion, comprising:

[0007] S1. Obtain a light field image containing multiple sub-aperture images, and convert the light field image into a macro-pixel image;

[0008] S2. Extract spatial features from the sub-aperture image and extract angle and limit features from the macro-pixel image to obtain spatial feature information, angle feature information and limit feature information;

[0009] S3. The spatial feature information, angular feature information, and limit feature information are fused and stacked to obtain the output features;

[0010] S4. Spatial upsampling of the output features yields a super-resolution sub-aperture image.

[0011] Optionally, the spatial feature information includes: a first correlation between pixels within the same sub-aperture image and a second correlation between pixels of multiple sub-aperture images;

[0012] The first correlation extraction uses an internal spatial feature extractor to convolve pixels within the same sub-aperture image; the internal spatial feature extractor includes a two-dimensional convolutional layer with a 3×3 kernel, a stride of 1, and a dilation factor of A, where A is equal to the angular resolution value of the light field image, and the internal spatial feature extractor uses zero padding during convolution.

[0013] The second correlation extraction employs a joint spatial feature extractor to convolve pixels between multiple sub-aperture images; wherein the joint spatial feature extractor includes a two-dimensional convolutional layer with a 3×3 kernel, a stride of 1, an inflation factor of 1, and zero padding during convolution.

[0014] Optionally, the angular feature information includes: the third correlation between pixels within the same macropixel image and the fourth correlation between pixels between different macropixel images;

[0015] The third correlation extraction uses an internal angle feature extractor to convolve pixels within the same macro-pixel image; the internal angle feature extractor includes a two-dimensional convolutional layer with a kernel of A×A, a stride of A, and a dilation factor of 1.

[0016] The fourth correlation extraction uses a joint angular feature extractor to convolve pixels between different macro-pixel images; the joint angular feature extractor includes a dilated convolution with a kernel size of 2A×2A, a stride size of 2A, and a dilation factor of 1.

[0017] Optionally, the limiting feature information includes: horizontal EPI features and vertical EPI features;

[0018] Horizontal EPI feature extraction employs a horizontal EPI feature extractor to convolve pixels in horizontally patched areas of the macro-pixel image; the horizontal EPI feature extractor includes a dilated convolution with a 1×A kernel. 2 The vertical movement step is 1, the horizontal movement step is A, and the horizontal EPI feature extractor does not perform zero padding during convolution.

[0019] Vertical EPI feature extraction employs a vertical EPI feature extractor that convolves pixels in vertical slices of the macro-pixel image; the vertical EPI feature extractor includes a dilated convolution with kernel A. 2 ×1, the horizontal movement step is 1, the vertical movement step is A, and the vertical EPI feature extractor does not perform zero padding during convolution;

[0020] Subsequently, the horizontal and vertical EPI features on the macropixel image were upsampled to obtain the limiting feature information.

[0021] Optionally, step S3 includes:

[0022] Spatial feature information, angular feature information, and limit feature information are concatenated in the horizontal and vertical directions to obtain the input feature X.

[0023] Average pooling is performed on the input feature X in both the horizontal and vertical directions to obtain two first-order statistics for each channel. These two first-order statistics include the average pooling value.

[0024]

[0025] Where h represents the number of sub-aperture image sequences in the vertical direction, and w represents the number of sub-aperture image sequences in the horizontal direction. X represents the h-th input feature. c Average pooling value in the horizontal direction, X represents the w-th input feature. c In the vertical direction, the average pooling value is given, where B represents the stride size of A, W represents the number of sub-aperture images in the horizontal direction, H represents the number of sub-aperture images in the vertical direction, and i represents the number of sub-aperture image sequences in the horizontal direction when performing average pooling on the input feature X. c (h,i) represents the h-th input feature value in the vertical direction and the i-th input feature value in the horizontal direction, and j represents the number of sub-aperture image sequences in the vertical direction when average pooling is performed on the input feature X. c (w,j) represents the w-th input feature value in the horizontal direction and the j-th input feature value in the vertical direction;

[0026] A gating mechanism is used to distribute attention to different channel features. The gating mechanism consists of two 1×1 convolutional layers and a ReLU activation function. First, the two first-order statistics obtained in the horizontal and vertical directions are concatenated in the channel dimension, and then the intermediate feature X1 is obtained by passing it through a shared 1×1 convolution.

[0027]

[0028] Where ReLU represents the activation function, W1 represents the convolutional layer weights of the gating mechanism, and * represents the convolution operation of the gating mechanism;

[0029] The intermediate feature X1 is split in the spatial dimension to obtain two identical tensors. Then, horizontal attention (att) is obtained through two 1×1 convolutional layers. h and vertical attention att w ;

[0030]

[0031] Where Sigmoid represents the activation function, W h Represents horizontal attention att h The weights W of the upper convolutional layer w Represents vertical attention att w The weights of the upper convolutional layer;

[0032] Finally, horizontal attention att h and vertical attention att w Applying this to the input feature X yields the output feature.

[0033]

[0034] Optionally, step S4 includes:

[0035] First, output features Remodeled from macropixel mode to sub-aperture mode;

[0036] Then use a 1×1 convolutional layer to reshape the output features The depth is increased to 2αC, where α represents the upgrade factor and C represents the number of channels;

[0037] Then, the depth-enhanced output features are transformed through a two-dimensional pixel transformation layer. Upsampled to αAH z ×αAW z Resolution, where H z W represents the height of the reshaped sub-aperture image. z Represents the width of the reshaped sub-aperture image;

[0038] Finally, a 1×1 convolutional layer is used to compress the channels to obtain a super-resolution sub-aperture image.

[0039] In summary, the present invention has at least the following beneficial technical effects:

[0040] 1. This invention continuously decomposes a given problem into two or more sub-problems of the same or related types. By designing different types of feature extractors, it extracts the spatial information, angular information and epipolar information of the four-dimensional light field image respectively. The above feature information is adaptively fused through an attention fusion mechanism, which significantly improves the quality and resolution of the sub-aperture image.

[0041] 2. By designing a specific type of two-dimensional convolution to achieve feature extraction, the computationally expensive three-dimensional convolution is effectively avoided. This not only reduces the complexity of the algorithm, but also makes the method proposed in this invention more efficient in practical applications, providing the possibility for real-time or large-scale data processing. Attached Figure Description

[0042] Figure 1 This is a flowchart of the spatial super-resolution method for light field images based on multi-feature fusion in an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of a spatial feature extractor.

[0044] Figure 3 This is a schematic diagram of an angle feature extractor.

[0045] Figure 4 This is a schematic diagram of the EPI feature extractor.

[0046] Figure 5 This is a schematic diagram of the channel attention module.

[0047] Figure 6 This is a schematic diagram of the spatial angle feature extraction and feature fusion process.

[0048] Figure 7 This is a schematic diagram of an optical field super-resolution network structure composed of multiple SAFE-Group modules. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0050] The terminology used in the following embodiments of the present invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification and appended claims of the present invention, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in the present invention refers to and includes any or all possible combinations of one or more of the listed items. The term “exemplary” means “serving as an example, embodiment, or illustration,” and any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. The terms “first” and “second” are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include one or more of that feature, and in the description of embodiments of the present invention, unless otherwise stated, “a plurality” means two or more.

[0051] This invention provides a spatial super-resolution method for light field images based on multi-feature fusion.

[0052] refer to Figure 1 A light field image spatial super-resolution method based on multi-feature fusion includes the following steps:

[0053] S1. Obtain a light field image containing multiple sub-aperture images and convert the light field image into a macro-pixel image.

[0054] A light field image containing multiple sub-aperture images is obtained. This light field image contains four-dimensional characteristics (position and orientation) and is also called a four-dimensional light field image. The four-dimensional light field image can be visualized as a macro-pixel image, represented by H × W macro pixels, where H represents the vertical number of sub-aperture images and W represents the horizontal number of sub-aperture images. By fixing (h, w), where h represents the number of sub-aperture image sequences in the vertical direction and w represents the number of sub-aperture image sequences in the horizontal direction, each macro-pixel can be represented as... Where U represents the row of the sub-aperture image and V represents the column of the sub-aperture image. Macro pixels are generated by arranging the pixels of the sub-aperture image according to their angular positions, and then the macro pixels are stitched together according to their spatial positions to obtain the macro pixel image.

[0055] S2. Perform spatial feature extraction on the sub-aperture image and angular and limit feature extraction on the macro-pixel image to obtain spatial feature information, angular feature information and limit feature information.

[0056] Spatial feature extraction employs spatial feature extractors, including the Spatial Interior Feature Extractor (SIFE) and the Joint Spatial Feature Extractor (SJFE). Angular feature extraction utilizes angular feature extractors, including the Angular Interior Feature Extractor (AIFE) and the Joint Angular Feature Extractor (AJFE). Limit feature extraction employs EPI feature extractors, including the Horizontal EPI Feature Extractor (H-EFE) and the Vertical EPI Feature Extractor (V-EFE).

[0057] Spatial feature information includes: the first correlation between pixels within the same sub-aperture image and the second correlation between pixels in multiple sub-aperture images.

[0058] Reference Figure 2 The first correlation extraction employs an internal spatial feature extractor that convolves pixels within the same sub-aperture image. This internal spatial feature extractor consists of a two-dimensional convolutional layer with a 3×3 kernel, a stride of 1, and a dilation factor of A, where A is equal to the angular resolution of the light field image. Furthermore, the internal spatial feature extractor uses zero padding during convolution. Zero padding ensures that the size of the input image remains unchanged.

[0059] The second correlation extraction employs a joint spatial feature extractor to convolve pixels across multiple sub-aperture images. This joint spatial feature extractor includes a two-dimensional convolutional layer with a 3×3 kernel, a stride of 1, and a dilation factor of 1. Furthermore, the joint spatial feature extractor uses zero padding during convolution. Zero padding ensures that the size of the input image remains unchanged.

[0060] Angular feature information includes: the third correlation between pixels within the same macropixel image and the fourth correlation between pixels in different macropixel images.

[0061] Reference Figure 3 The third correlation extraction uses an internal angle feature extractor to convolve pixels within the same macro-pixel image; the internal angle feature extractor includes a two-dimensional convolutional layer with a kernel of A×A, a stride of A, and a dilation factor of 1.

[0062] The fourth correlation extraction uses a joint angle feature extractor to convolve pixels between different macro-pixel images; the joint angle feature extractor includes a dilated convolution with a kernel size of 2A×2A, a stride size of 2A, and a dilation factor of 1; at the same time, in order to maintain the parallax structure of the light field image, a two-dimensional pixel transformation layer is used to ensure that the size of the output feature map does not change.

[0063] Limiting feature information includes: horizontal EPI features and vertical EPI features;

[0064] Reference Figure 4 The extraction of horizontal EPI features employs a horizontal EPI feature extractor that convolves the pixels of horizontally patched areas on the macro-pixel image; the horizontal EPI feature extractor includes a dilated convolution with a 1×A kernel. 2 The vertical movement step is 1, the horizontal movement step is A, and the horizontal EPI feature extractor does not perform zero padding during convolution to ensure that the output image size does not change.

[0065] Vertical EPI feature extraction employs a vertical EPI feature extractor that convolves pixels in vertical slices of the macro-pixel image; the vertical EPI feature extractor includes a dilated convolution with kernel A. 2 ×1, with a horizontal step size of 1 and a vertical step size of A, and the vertical EPI feature extractor does not perform zero padding during convolution.

[0066] Subsequently, the horizontal and vertical EPI features on the macropixel image are upsampled to obtain the limiting feature information, while ensuring that the size of the output image remains unchanged.

[0067] S3. The spatial feature information, angular feature information, and limit feature information are fused and stacked to obtain the output features.

[0068] refer to Figure 5 The channel attention module is used to concatenate the first, second, third, and fourth correlations along the channel dimension to obtain the input feature X.

[0069] Average pooling is performed on the input feature X in both the horizontal and vertical directions to obtain two first-order statistics for each channel. These two first-order statistics include the average pooling value.

[0070]

[0071] Where h represents the number of sub-aperture image sequences in the vertical direction, and w represents the number of sub-aperture image sequences in the horizontal direction. X represents the h-th input feature. c Average pooling value in the horizontal direction, X represents the w-th input feature. c In the vertical direction, the average pooling value is given, where B represents the stride size of A, W represents the number of sub-aperture images in the horizontal direction, H represents the number of sub-aperture images in the vertical direction, and i represents the number of sub-aperture image sequences in the horizontal direction when performing average pooling on the input feature X. c (h,i) represents the h-th input feature value in the vertical direction and the i-th input feature value in the horizontal direction, and j represents the number of sub-aperture image sequences in the vertical direction when average pooling is performed on the input feature X. c (w,j) represents the w-th input feature value in the horizontal direction and the j-th input feature value in the vertical direction.

[0072] A gating mechanism is employed to distribute attention to different channel features. This gating mechanism consists of two 1×1 convolutional layers and a ReLU activation function. First, the two first-order statistics obtained in the horizontal and vertical directions are concatenated along the channel dimension. Then, a shared 1×1 convolution is used to obtain the intermediate feature X1, as shown in the following expression:

[0073]

[0074] Where ReLU represents the activation function, W1 represents the weights of the gated convolutional layer, and * represents the convolution operation of the gated mechanism.

[0075] The intermediate feature X1 is split in the spatial dimension to obtain two identical tensors. Then, horizontal attention (att) is obtained through two 1×1 convolutional layers. h and vertical attention att w ;

[0076]

[0077] Where Sigmoid represents the activation function, W h Represents horizontal attention att h The weights W of the upper convolutional layer w Represents vertical attention att w The weights of the upper convolutional layer;

[0078] Finally, horizontal attention att h and vertical attention att w Applying this to the input feature X yields the output feature.

[0079]

[0080] Reference Figure 6 , Figure 6 This demonstrates the process of extracting spatial feature information, angular feature information, epipolar feature information, as well as feature fusion and stacking in steps S2-S3, ultimately yielding the output features. Figure 6 In this module, SAEF-Block is the spatial angle epipolar feature extraction module, and the Attention module is the feature fusion module. The feature fusion module allows channel features to be adaptively rescaled at multiple levels, highlighting important features while suppressing unnecessary ones. SIFE, SJFE, AIFE, AJFE, H-EFE, and V-EFE represent the spatial interior feature extractor, joint spatial feature extractor, angle interior feature extractor, joint angle feature extractor, horizontal EPI feature extractor, and vertical EPI feature extractor, respectively, used to extract the first correlation, second correlation, third correlation, fourth correlation, horizontal EPI features, and vertical EPI features. These extracted features are then concatenated and input into the feature fusion module for feature fusion, resulting in the output feature.

[0081] Reference Figure 7 , Figure 7 This paper demonstrates the process of constructing a spatial super-resolution network (LFSSRNet) using the SAFE-Group module. Specifically, LFSSRNet is designed by stacking spatial convolution and angular convolution, which outperforms existing methods on multiple light field datasets. Figure 7The system comprises five SAFE-Group modules and an Upsampling Module. Each SAFE-Group module includes four sequentially connected SAEF-Block modules. This configuration aims to accurately extract image features. A residual structure is established between each subsequent SAFE-Group module and the preceding SAFE-Group module, resulting in a total of three residual structures. This configuration effectively learns high-frequency information from the residuals in the low-resolution image, leading to a clearer and more realistic reconstructed structure. The summation of the last residual structure outputs the low-resolution image features. These low-resolution image features are then input into the Upsampling Module for convolution, upsampling, and further convolution operations. This improves the spatial resolution of the light field features, resulting in a high spatial resolution light field feature map. Finally, the high spatial resolution light field feature map is bilinearly interpolated with the initial output image (LR MacPI) to obtain a high-resolution sub-aperture image (HR SAI).

[0082] S4. Spatial upsampling of the output features yields a super-resolution sub-aperture image.

[0083] First, output features The macro-pixel mode is reshaped into a sub-aperture mode; then a 1×1 convolutional layer is used to reshape the output features. The depth is increased to 2αC, where y represents the upgrade factor and C represents the number of channels; then, the output features with increased depth are transformed through a two-dimensional pixel transformation layer. Upsampled to αAH z ×αAW z Resolution, where H z W represents the height of the reshaped sub-aperture image. z This represents the width of the reconstructed sub-aperture image; finally, a 1×1 convolutional layer is used to compress the channels to obtain a super-resolution sub-aperture image.

[0084] In summary, the proposed light field image spatial super-resolution method based on multi-feature fusion continuously decomposes a given problem into two or more sub-problems of the same or related types. By designing different types of feature extractors, it extracts spatial, angular, and epipolar information from the four-dimensional light field image, and adaptively fuses these feature information through an attention fusion mechanism. A light field spatial super-resolution network, LFSSRNet, was designed using stacked spatial and angular convolutions, and its performance on multiple light field datasets outperforms existing methods. Using the proposed method can significantly improve the quality and resolution of sub-aperture images, introducing an innovative approach to content generation for glasses-free 3D technology. This technological breakthrough makes 8K and even 16K resolution 3D screen display content a reality, greatly promoting the development of large-screen glasses-free 3D display technology. This advancement not only improves the clarity and immersion of the visual experience but also lays a solid foundation for future glasses-free 3D displays in larger-scale and higher-precision applications. Furthermore, by designing specific types of two-dimensional convolutions to achieve feature extraction, it effectively avoids the use of computationally expensive three-dimensional convolutions. This not only reduces the complexity of the algorithm, but also makes the method proposed in this invention more efficient in practical applications, making real-time or large-scale data processing possible.

[0085] The above description of the embodiments is only used to provide a detailed introduction to the technical solution of the present invention. However, the description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention, and should not be construed as a limitation of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for spatial super-resolution of light field images based on multi-feature fusion, characterized in that, The method comprises the following steps: S1, acquiring a light field image containing a plurality of sub-aperture images, and converting the light field image into a macro-pixel image; S2, performing spatial feature extraction on the sub-aperture images, and performing angle feature extraction and limit feature extraction on the macro-pixel image to obtain spatial feature information, angle feature information and limit feature information; The spatial feature information comprises a first correlation between pixels in the same sub-aperture image and a second correlation between pixels of a plurality of sub-aperture images; The extraction of the first correlation adopts an internal spatial feature extractor to convolve pixels inside the same sub-aperture image; wherein the internal spatial feature extractor comprises a two-dimensional convolution layer, the convolution kernel is 3*3, the moving step is 1, and the inflation factor is A wherein, A is equal to the angle resolution value of the light field image, and the internal spatial feature extractor uses zero padding when convolution. The extraction of the second correlation adopts a joint spatial feature extractor to perform convolution on the pixels between the plurality of sub-aperture images; wherein the joint spatial feature extractor comprises a two-dimensional convolution layer, the convolution kernel is 3*3, the moving step is 1, the inflation factor is 1, and the joint spatial feature extractor uses zero padding when performing convolution; The angle feature information comprises a third correlation between pixels in the same macro-pixel image and a fourth correlation between pixels of different macro-pixel images; The third correlation is extracted by using an internal angle feature extractor to convolve the pixels inside the same macro-pixel image; wherein the internal angle feature extractor comprises a two-dimensional convolution layer, the convolution kernel is A × A , the moving step is A , and the expansion factor is 1; The fourth correlation is extracted by using a joint angle feature extractor to convolve pixels between different macro-pixel images; wherein the joint angle feature extractor includes a hollow convolution, the convolution kernel is 2 A ×2 A , the moving step is 2 A , and the expansion factor is 1; The limit feature information comprises horizontal EPI features and vertical EPI features; The extraction of the horizontal EPI feature adopts a horizontal EPI feature extractor to perform convolution on the pixels of the horizontal patches on the macro-pixel image; wherein the horizontal EPI feature extractor comprises a hole convolution, and the convolution kernel is 1x A 2 , the moving step in the vertical direction is 1, the moving step in the horizontal direction is A , and the horizontal EPI feature extractor does not perform zero padding when performing convolution. The vertical EPI feature extraction adopts a vertical EPI feature extractor to perform convolution on the pixels of the vertical slice on the macro-pixel image; wherein the vertical EPI feature extractor comprises a hollow convolution, and the convolution kernel is A 2 , the horizontal direction moving step is 1, the vertical direction moving step is A , and the vertical EPI feature extractor does not perform zero padding when performing convolution. Then, the horizontal EPI features and the vertical EPI features on the macro-pixel image are respectively subjected to up-sampling processing to obtain the limit feature information; S3, fusing and stacking the spatial feature information, the angle feature information and the limit feature information to obtain output features, specifically: The spatial feature information, the angle feature information and the limit feature information are spliced in horizontal and vertical directions to obtain input features ; average pooling the input features in horizontal and vertical directions respectively to obtain two first order statistics of channels, the two first order statistics including average pooling values , , ; wherein, a number of sub-aperture images in the vertical direction, a number of sub-aperture images in the horizontal direction, representing the first input feature in the horizontal direction, representing the first input feature in the vertical direction, representing a moving step and having a size of , representing a number of sub-aperture images in the horizontal direction, representing a number of sub-aperture images in the vertical direction, representing a number of sub-aperture images in the horizontal direction when performing average pooling on the input feature , representing the first input feature value in the vertical direction and the first input feature value in the horizontal direction, representing a number of sub-aperture images in the vertical direction when performing average pooling on the input feature , representing the first input feature value in the horizontal direction and the first input feature value in the vertical direction. The gating mechanism is composed of two 1x1 convolution layers and an activation function; first, the two first-order statistics obtained in the horizontal and vertical directions are spliced in the channel dimension, and then an intermediate feature is obtained through a shared 1x1 convolution . ​ wherein, represents an activation function, represents a convolution layer weight of a gating mechanism, represents a convolution operation of a gating mechanism; Split in spatial dimension, get two same tensors Split in spatial dimension, get two same tensors Then get horizontal attention by two 1x1 convolution layers And vertical attention ; wherein, represents an activation function, represents a horizontal attention weights of the upper convolutional layer, represents a vertical attention weights of the upper convolutional layer; Finally, horizontal attention and vertical attention are applied to the input features to obtain output features : ； S4, performing spatial up-sampling on the output features to obtain a super-resolution sub-aperture image. 2.The method of claim 1, wherein, The step S4 comprises: First, output features reconstruction from macro-pixel mode to sub-aperture mode; The reshaped output features are again passed through a 1x1 convolutional layer The depth is increased to 2 wherein, represents an upscaling factor, represents the number of channels; The output feature with increased depth is then upsampled by a two-dimensional pixel transformation layer upsampled to z × z resolution, wherein, z represents a high of the remolded sub-aperture image, z represents a width of the remolded sub-aperture image; Finally, a 1*1 convolution layer is adopted to compress the channels to obtain a super-resolution sub-aperture image.

Images