Light field image space super-resolution method based on multi-feature fusion

Abstract

The invention relates to the technical field of image resolution technology, and discloses a light field image space super-resolution method based on multi-feature fusion, which comprises the following steps: acquiring a light field image comprising a plurality of sub-aperture images, and converting the light field image into a macro-pixel image; spatial feature extraction is carried out on the sub-aperture images, angle feature extraction and limit feature extraction are carried out on the macro-pixel images, and spatial feature information, angle feature information and limit feature information are obtained; fusing and stacking the spatial feature information, the angle feature information and the limit feature information to obtain output features; performing spatial up-sampling on the output features to obtain a super-resolution sub-aperture image; according to the method, the reconstruction resolution of the light field image is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image resolution technology, and in particular to a light field image spatial super-resolution method based on multi-feature fusion. Background Art

[0002] The light field camera contains a microlens array, which can record light passing through it in different directions, so that the light field camera can take light field photos and obtain many photos at different angles and positions when taking a light field photo in the later refocusing process. Therefore, it is widely used in depth estimation, 3D reconstruction, object segmentation and image refocusing.

[0003] Due to the spatial limitations of the microlens array, the spatial resolution of the sub-aperture image of the light field image is much lower than the resolution of the two-dimensional image captured by traditional imaging devices. Therefore, how to improve the spatial resolution of the light field image has become a research hotspot in the field of image resolution technology.

[0004] In related technologies, subaperture images or epipolar images (EPI) are often stacked at different angles to reconstruct high-resolution light field images. However, due to the complex four-dimensional nature of light field images, the above two methods cannot fully utilize the inherent angular and spatial information in the data, resulting in unsatisfactory reconstruction resolution. Summary of the invention

[0005] In order to improve the reconstruction resolution of light field images, the present invention provides a light field image spatial super-resolution method based on multi-feature fusion.

[0006] In a first aspect, the present invention provides a light field image spatial super-resolution method based on multi-feature fusion, comprising:

[0007] S1, acquiring a light field image including a plurality of sub-aperture images, and converting the light field image into a macro-pixel image;

[0008] S2, performing spatial feature extraction on the sub-aperture image, and performing angle feature extraction and limit feature extraction on the macro-pixel image to obtain spatial feature information, angle feature information and limit feature information;

[0009] S3, fusing and stacking the spatial feature information, the angle feature information, and the limit feature information to obtain output features;

[0010] S4. Perform spatial upsampling on the output features to obtain a super-resolution sub-aperture image.

[0011] Optionally, the spatial feature information includes: a first correlation between pixels within a same sub-aperture image and a second correlation between pixels of a plurality of sub-aperture images;

[0012] The first correlation is extracted by convolving pixels inside the same sub-aperture image using an internal spatial feature extractor; wherein the internal spatial feature extractor includes a two-dimensional convolution layer, a convolution kernel of 3×3, a moving step of 1, and a dilation factor of A, wherein A is equal to the angular resolution value of the light field image, and the internal spatial feature extractor uses zero padding during convolution;

[0013] The second correlation is extracted by convolving pixels between multiple sub-aperture images using a joint spatial feature extractor; wherein the joint spatial feature extractor includes a two-dimensional convolution layer, a convolution kernel of 3×3, a moving step of 1, a dilation factor of 1, and the joint spatial feature extractor uses zero padding during convolution.

[0014] Optionally, the angle feature information includes: a third correlation between pixels within the same macro-pixel image and a fourth correlation between pixels between different macro-pixel images;

[0015] The third correlation is extracted by convolving pixels inside the same macro-pixel image using an internal angle feature extractor; wherein the internal angle feature extractor includes a two-dimensional convolution layer, a convolution kernel of A×A, a moving step of A, and a dilation factor of 1;

[0016] The fourth correlation is extracted by convolving pixels between different macro-pixel images using a joint angle feature extractor; wherein the joint angle feature extractor includes a dilated convolution with a convolution kernel of 2A×2A, a moving step of 2A, and a dilation factor of 1.

[0017] Optionally, the extreme feature information includes: horizontal EPI features and vertical EPI features;

[0018] The horizontal EPI feature extraction uses a horizontal EPI feature extractor to convolve the pixels of the horizontal patch on the macro pixel image; the horizontal EPI feature extractor includes a dilated convolution with a convolution kernel of 1×A 2 , the vertical movement step size is 1, the horizontal movement step size is A, and the horizontal EPI feature extractor does not perform zero padding during convolution;

[0019] The vertical EPI feature extraction uses a vertical EPI feature extractor to convolve the pixels of the vertical slice on the macro-pixel image; wherein the vertical EPI feature extractor includes a dilated convolution with a convolution kernel of A 2 ×1, the horizontal movement step is 1, the vertical movement step is A, and the vertical EPI feature extractor does not perform zero padding during convolution;

[0020] Afterwards, the horizontal EPI features and the vertical EPI features on the macro-pixel image are upsampled respectively to obtain the extreme feature information.

[0021] Optionally, step S3 includes:

[0022] The spatial feature information, the angle feature information, and the limit feature information are concatenated in the horizontal and vertical directions to obtain the input feature X;

[0023] The input feature X is averaged and pooled in the horizontal and vertical directions respectively to obtain two first-order statistics of the channel. The two first-order statistics include the average pooling value

[0024]

[0025] Where h represents the number of sub-aperture image sequences in the vertical direction, and w represents the number of sub-aperture image sequences in the horizontal direction. Represents the hth input feature X c The average pooling value in the horizontal direction, Represents the w-th input feature X c The average pooling value in the vertical direction, B represents the moving step size and the size is A, W represents the number of sub-aperture images in the horizontal direction, H represents the number of sub-aperture images in the vertical direction, i represents the number of sub-aperture images in the horizontal direction when the input feature X is averaged pooled in the horizontal direction, X c (h,i) represents the hth input feature value in the vertical direction and the ith input feature value in the horizontal direction. j represents the number of sub-aperture image sequences in the vertical direction when the input feature X is average pooled in the vertical direction. c (w,j) represents the w-th input feature value in the horizontal direction and the j-th input feature value in the vertical direction;

[0026] A gating mechanism is used to allocate attention to different channel features. The gating mechanism consists of two 1×1 convolutional layers and a Relu activation function. First, the two first-order statistics obtained in the horizontal and vertical directions are concatenated in the channel dimension, and then a shared 1×1 convolution is performed to obtain the intermediate feature X1.

[0027]

[0028] Among them, Relu represents the activation function, W1 represents the convolution layer weight of the gating mechanism, and * represents the convolution operation of the gating mechanism;

[0029] Split the intermediate feature X1 in the spatial dimension to obtain two identical tensors Then two 1×1 convolutional layers are used to obtain the horizontal attention att h and vertical attention att w ;

[0030]

[0031] Among them, Sigmoid represents the activation function, W h Represents the horizontal attention att h The weight of the upper convolutional layer, W w Represents vertical attention att w The weights of the previous convolutional layer;

[0032] Finally, the horizontal attention att h and vertical attention att w Acting on the input feature X, the output feature is obtained

[0033]

[0034] Optionally, step S4 includes:

[0035] First, the output features Reshape from macro-pixel mode to sub-aperture mode;

[0036] Then use a 1×1 convolution layer to reshape the output features The depth is increased to 2αC, where α represents the upgrade factor and C represents the number of channels;

[0037] Then, the output features with increased depth are transformed through a two-dimensional pixel transformation layer Upsampling to αAH z ×αAW z Resolution, where H z represents the height of the reshaped sub-aperture image, W z Represents the width of the sub-aperture image after reshaping;

[0038] Finally, a 1×1 convolutional layer is used to compress the channels to obtain a super-resolution sub-aperture image.

[0039] In summary, the present invention at least includes the following beneficial technical effects:

[0040] 1. The present invention continuously decomposes a given problem into two or more sub-problems of the same or related types, and respectively extracts the spatial information, angle information and epipolar information of the four-dimensional light field image by designing different types of feature extractors, and adaptively fuses the above feature information through the attention fusion mechanism, thereby significantly improving the quality and resolution of the sub-aperture image;

[0041] 2. By designing a specific type of two-dimensional convolution to achieve feature extraction, the use of computationally expensive three-dimensional convolution is effectively avoided. This not only reduces the complexity of the algorithm, but also makes the method proposed in the present invention more efficient in practical applications, providing the possibility for real-time or large-scale data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of a light field image spatial super-resolution method based on multi-feature fusion in an embodiment of the present invention.

[0043] Figure 2 Schematic diagram of the spatial feature extractor.

[0044] Figure 3 Schematic diagram of the angle feature extractor.

[0045] Figure 4 Schematic diagram of the EPI feature extractor.

[0046] Figure 5 Schematic diagram of the channel attention module.

[0047] Figure 6 Schematic diagram of spatial angle feature extraction and feature fusion process.

[0048] Figure 7 Schematic diagram of the light field super-resolution network structure composed of multiple SAFE-Group modules. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] The terms used in the following embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to be used as limitations of the present invention. As used in the specification of the present invention and the appended claims, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear contrary indication in its context. It should also be understood that the term "and / or" used in the present invention refers to and includes any or all possible combinations of one or more listed items. The term "exemplary" means "used as an example, embodiment or illustrative", and any embodiment described as "exemplary" here does not have to be interpreted as being superior to or better than other embodiments. The terms "first" and "second" are only used for descriptive purposes, and cannot be understood as implying or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" can expressly or implicitly include one or more of these features, and in the description of the embodiments of the present invention, unless otherwise specified, the meaning of "multiple" is two or more.

[0051] An embodiment of the present invention provides a light field image spatial super-resolution method based on multi-feature fusion.

[0052] refer to Figure 1 , a light field image spatial super-resolution method based on multi-feature fusion, comprising the following steps:

[0053] S1. Acquire a light field image including a plurality of sub-aperture images, and convert the light field image into a macro-pixel image.

[0054] A light field image containing multiple sub-aperture images is obtained. The light field image contains four-dimensional characteristics (position and direction). The light field image is also called a four-dimensional light field image. The four-dimensional light field image can be visualized in the form of a macro-pixel image, which can be represented as H×W macro-pixels, where H represents the vertical number of sub-aperture images and W represents the horizontal number of sub-aperture images. By fixing (h, w), where h represents the number of sub-aperture image sequences in the vertical direction and w represents the number of sub-aperture image sequences in the horizontal direction. Each macro-pixel can be represented as Where U represents the row of the sub-aperture image, and V represents the column of the sub-aperture image. Macro pixels are generated by arranging the pixels of the sub-aperture image according to their angular positions, and then they are spliced ​​according to the spatial positions of the macro pixels to obtain a macro pixel image.

[0055] S2. Perform spatial feature extraction on the sub-aperture image, and perform angle feature extraction and limit feature extraction on the macro-pixel image to obtain spatial feature information, angle feature information and limit feature information.

[0056] Among them, the spatial feature extraction adopts the spatial feature extractor, which includes the spatial internal feature extractor (SIFE) and the joint spatial feature extractor (SJFE). The angle feature extraction adopts the angle feature extractor, which includes the angle internal feature extractor (AIFE) and the joint angle feature extractor (AJFE). The extreme feature extraction adopts the EPI feature extractor, which includes the horizontal EPI feature extractor (H-EFE) and the vertical EPI feature extractor (V-EFE).

[0057] The spatial feature information includes: a first correlation between pixels within the same sub-aperture image and a second correlation between pixels of multiple sub-aperture images.

[0058] Reference Figure 2 , the first correlation is extracted by convolving the pixels inside the same sub-aperture image using an internal spatial feature extractor; wherein the internal spatial feature extractor includes a two-dimensional convolution layer, a convolution kernel of 3×3, a moving step of 1, and a dilation factor of A, wherein A is equal to the angular resolution value of the light field image, and the internal spatial feature extractor uses zero padding during convolution. Using zero padding can ensure that the size of the input image does not change.

[0059] The second correlation is extracted by convolving pixels between multiple sub-aperture images using a joint spatial feature extractor; wherein the joint spatial feature extractor includes a two-dimensional convolution layer, a convolution kernel of 3×3, a moving step of 1, a dilation factor of 1, and the joint spatial feature extractor uses zero padding during convolution. Using zero padding can ensure that the size of the input image does not change.

[0060] The angle feature information includes: a third correlation between pixels within the same macro-pixel image and a fourth correlation between pixels between different macro-pixel images.

[0061] Reference Figure 3 The third correlation is extracted by convolving pixels inside the same macro-pixel image using an internal angle feature extractor; wherein the internal angle feature extractor includes a two-dimensional convolution layer with a convolution kernel of A×A, a moving step of A, and a dilation factor of 1.

[0062] The fourth correlation is extracted by convolving pixels between different macro-pixel images using a joint angular feature extractor; the joint angular feature extractor includes a dilated convolution with a convolution kernel of 2A×2A, a moving step of 2A, and a dilation factor of 1; at the same time, in order to maintain the parallax structure of the light field image, a two-dimensional pixel transformation layer is used to ensure that the size of the output feature map does not change.

[0063] The extreme feature information includes: horizontal EPI features and vertical EPI features;

[0064] Reference Figure 4 , the horizontal EPI feature extraction uses a horizontal EPI feature extractor to convolve the pixels of the horizontal patch on the macro pixel image; wherein the horizontal EPI feature extractor includes a dilated convolution with a convolution kernel of 1×A 2 , the vertical moving step size is 1, the horizontal moving step size is A, and the horizontal EPI feature extractor does not perform zero padding during convolution to ensure that the output image size does not change.

[0065] The vertical EPI feature extraction uses a vertical EPI feature extractor to convolve the pixels of the vertical slice on the macro-pixel image; wherein the vertical EPI feature extractor includes a dilated convolution with a convolution kernel of A 2 ×1, the horizontal moving step is 1, the vertical moving step is A, and the vertical EPI feature extractor does not perform zero padding during convolution.

[0066] Afterwards, the horizontal EPI features and the vertical EPI features on the macro-pixel image are upsampled respectively to obtain the extreme feature information and ensure that the output image size does not change.

[0067] S3. Fuse and stack the spatial feature information, angle feature information, and limit feature information to obtain output features.

[0068] refer to Figure 5 , the channel attention module is used to concatenate the first correlation, the second correlation, the third correlation and the fourth correlation in the channel dimension to obtain the input feature X;

[0069] The input feature X is averaged and pooled in the horizontal and vertical directions respectively to obtain two first-order statistics of the channel. The two first-order statistics include the average pooling value

[0070]

[0071] Where h represents the number of sub-aperture image sequences in the vertical direction, and w represents the number of sub-aperture image sequences in the horizontal direction. Represents the hth input feature X c The average pooling value in the horizontal direction, Represents the w-th input feature X c The average pooling value in the vertical direction, B represents the moving step size and the size is A, W represents the number of sub-aperture images in the horizontal direction, H represents the number of sub-aperture images in the vertical direction, i represents the number of sub-aperture images in the horizontal direction when the input feature X is averaged pooled in the horizontal direction, X c (h,i) represents the hth input feature value in the vertical direction and the ith input feature value in the horizontal direction. j represents the number of sub-aperture image sequences in the vertical direction when the input feature X is average pooled in the vertical direction. c (w,j) represents the w-th input feature value in the horizontal direction and the j-th input feature value in the vertical direction.

[0072] A gating mechanism is used to allocate attention to different channel features. The gating mechanism consists of two 1×1 convolutional layers and a Relu activation function. First, the two first-order statistics obtained in the horizontal and vertical directions are concatenated in the channel dimension, and then a shared 1×1 convolution is performed to obtain the intermediate feature X1. The expression is as follows:

[0073]

[0074] Among them, Relu represents the activation function, W1 represents the weight of the convolution layer of the gating mechanism, and * represents the convolution operation of the gating mechanism.

[0075] Split the intermediate feature X1 in the spatial dimension to obtain two identical tensors Then two 1×1 convolutional layers are used to obtain the horizontal attention att h and vertical attention att w ;

[0076]

[0077] Among them, Sigmoid represents the activation function, W h Represents the horizontal attention att h The weight of the upper convolutional layer, W w Represents vertical attention att w The weights of the previous convolutional layer;

[0078] Finally, the horizontal attention att h and vertical attention att w Acting on the input feature X, the output feature is obtained

[0079]

[0080] Reference Figure 6 , Figure 6 The process of extracting spatial feature information, angular feature information, and epipolar feature information in steps S2-S3, as well as feature fusion and stacking, is shown, and the output feature is finally obtained. Figure 6 The SAEF-Block in it is the spatial angle polar feature extraction module, and the Attention module is the feature fusion module. That is, through the feature fusion module, the channel features can be adaptively rescaled at multiple levels, highlighting important features while suppressing unnecessary features. Among them, SIFE, SJFE, AIFE, AJFE, H-EFE, and V-EFE represent the spatial internal feature extractor, the joint spatial feature extractor, the angle internal feature extractor, the joint angle feature extractor, the horizontal EPI feature extraction, and the vertical EPI feature extractor, respectively, which are used to extract the first correlation, the second correlation, the third correlation, the fourth correlation, the horizontal EPI feature, and the vertical EPI feature, respectively, and the above extracted features are spliced ​​and input into the feature fusion module for feature fusion to obtain the output feature.

[0081] Reference Figure 7 , Figure 7 The process of using the SAFE-Group module to build a light field spatial super-resolution network (LFSSRNet network) is demonstrated. That is, the light field spatial super-resolution network LFSSRNet is designed by stacking spatial convolution and angular convolution, making its performance on multiple light field datasets better than existing methods; Figure 7The method includes five SAFE-Group modules and Upsamping Module modules, wherein each SAFE-Group module includes four SAEF-Block modules connected in sequence, and the purpose of such setting is to accurately extract image features; and a residual structure is set between the latter SAFE-Group module and the previous SAFE-Group module, and a total of three groups of residual structures are set, and the purpose of such setting is to effectively learn the residual of high-frequency information from the low-resolution image, so that the reconstructed structure is clearer and more realistic; at the same time, the result of the summation output of the last residual structure is the low-resolution image feature; then, the low-resolution image feature is input into the Upsamping Module module for convolution operation, upsampling operation and convolution operation, and the purpose is to improve the spatial resolution of the light field feature, thereby outputting a light field feature map with high spatial resolution; finally, the light field feature map with high spatial resolution is bilinearly interpolated (BilinearUpsamping) with the initial output image, i.e., the low-resolution macro-pixel image (LR MacPI), to obtain a high-resolution sub-aperture image (HR SAI).

[0082] S4. Perform spatial upsampling on the output features to obtain a super-resolution sub-aperture image.

[0083] First, the output features Reshape from macro-pixel mode to sub-aperture mode; then use a 1×1 convolution layer to transform the reshaped output features The depth is increased to 2αC, where y represents the upgrade factor and C represents the number of channels; then the output features after the depth increase are transformed through the two-dimensional pixel transformation layer Upsampling to αAH z ×αAW z Resolution, where H z represents the height of the reshaped sub-aperture image, W z represents the width of the reshaped sub-aperture image; finally, a 1×1 convolutional layer is used to compress the channel to obtain a super-resolution sub-aperture image.

[0084] In summary, the light field image spatial super-resolution method based on multi-feature fusion proposed in the present invention continuously decomposes a given problem into two or more sub-problems of the same or related types, and extracts spatial information, angle information and epipolar information of four-dimensional light field images respectively by designing different types of feature extractors, and adaptively fuses the above feature information through the attention fusion mechanism. The light field spatial super-resolution network LFSSRNet is designed by stacking spatial convolution and angle convolution, and its performance on multiple light field data sets is better than the existing methods. The method proposed in the present invention can significantly improve the quality and resolution of sub-aperture images, and introduces an innovative method for naked-eye 3D technology in content generation. This technological breakthrough makes 3D screen display content with 8K or even 16K resolution a reality, and greatly promotes the development of large-screen naked-eye 3D display technology. This progress not only improves the clarity and immersion of the visual experience, but also lays a solid foundation for future naked-eye 3D displays in larger-scale and higher-precision applications. Secondly, feature extraction is achieved by designing a specific type of two-dimensional convolution, which effectively avoids the use of three-dimensional convolution with high computational cost. This not only reduces the complexity of the algorithm, but also makes the method proposed in the present invention more efficient in practical applications, providing the possibility for real-time or large-scale data processing.

[0085] As described above, the above embodiments are only used to introduce the technical solution of the present invention in detail, but the description of the above embodiments is only used to help understand the method and core idea of ​​the present invention and should not be understood as limiting the present invention. Changes or substitutions that can be easily thought of by technicians in this technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A light field image spatial super-resolution method based on multi-feature fusion, characterized in that: include: S1, acquiring a light field image including a plurality of sub-aperture images, and converting the light field image into a macro-pixel image; S2, performing spatial feature extraction on the sub-aperture image, and performing angle feature extraction and limit feature extraction on the macro-pixel image to obtain spatial feature information, angle feature information and limit feature information; S3, fusing and stacking the spatial feature information, the angle feature information, and the limit feature information to obtain output features; S4. Perform spatial upsampling on the output features to obtain a super-resolution sub-aperture image.

2. The light field image spatial super-resolution method based on multi-feature fusion according to claim 1, characterized in that: The spatial feature information includes: a first correlation between pixels within the same sub-aperture image and a second correlation between pixels of multiple sub-aperture images; The first correlation is extracted by convolving pixels inside the same sub-aperture image using an internal spatial feature extractor; wherein the internal spatial feature extractor includes a two-dimensional convolution layer, a convolution kernel of 3×3, a moving step of 1, and a dilation factor of A, wherein A is equal to the angular resolution value of the light field image, and the internal spatial feature extractor uses zero padding during convolution; The second correlation is extracted by convolving pixels between multiple sub-aperture images using a joint spatial feature extractor; wherein the joint spatial feature extractor includes a two-dimensional convolution layer, a convolution kernel of 3×3, a moving step of 1, a dilation factor of 1, and the joint spatial feature extractor uses zero padding during convolution.

3. The light field image spatial super-resolution method based on multi-feature fusion according to claim 2, characterized in that: The angle feature information includes: a third correlation between pixels within the same macro-pixel image and a fourth correlation between pixels in different macro-pixel images; The third correlation is extracted by convolving pixels inside the same macro-pixel image using an internal angle feature extractor; wherein the internal angle feature extractor includes a two-dimensional convolution layer, a convolution kernel of A×A, a moving step of A, and a dilation factor of 1; The fourth correlation is extracted by convolving pixels between different macro-pixel images using a joint angle feature extractor; wherein the joint angle feature extractor includes a dilated convolution with a convolution kernel of 2A×2A, a moving step of 2A, and a dilation factor of 1.

4. The light field image spatial super-resolution method based on multi-feature fusion according to claim 3, characterized in that: The extreme feature information includes: horizontal EPI features and vertical EPI features; The horizontal EPI feature extraction uses a horizontal EPI feature extractor to convolve the pixels of the horizontal patch on the macro pixel image; wherein the horizontal EPI feature extractor includes a dilated convolution with a convolution kernel of 1×A 2 , the vertical movement step size is 1, the horizontal movement step size is A, and the horizontal EPI feature extractor does not perform zero padding during convolution; The vertical EPI feature extraction uses a vertical EPI feature extractor to convolve the pixels of the vertical slice on the macro-pixel image; wherein the vertical EPI feature extractor includes a dilated convolution with a convolution kernel of A 2 ×1, the horizontal movement step is 1, the vertical movement step is A, and the vertical EPI feature extractor does not perform zero padding during convolution; Afterwards, the horizontal EPI features and the vertical EPI features on the macro-pixel image are upsampled respectively to obtain the extreme feature information.

5. The light field image spatial super-resolution method based on multi-feature fusion according to claim 4, characterized in that: The step S3 comprises: The spatial feature information, the angle feature information, and the limit feature information are concatenated in the horizontal and vertical directions to obtain the input feature X; The input feature X is averaged and pooled in the horizontal and vertical directions respectively to obtain two first-order statistics of the channel. The two first-order statistics include the average pooling value Where h represents the number of sub-aperture image sequences in the vertical direction, and w represents the number of sub-aperture image sequences in the horizontal direction. Represents the hth input feature X c The average pooling value in the horizontal direction, Represents the w-th input feature X c The average pooling value in the vertical direction, B represents the moving step size and the size is A, W represents the number of sub-aperture images in the horizontal direction, H represents the number of sub-aperture images in the vertical direction, i represents the number of sub-aperture images in the horizontal direction when the input feature X is averaged pooled in the horizontal direction, X c (h,i) represents the hth input feature value in the vertical direction and the ith input feature value in the horizontal direction. j represents the number of sub-aperture image sequences in the vertical direction when the input feature X is average pooled in the vertical direction. c (w,j) represents the w-th input feature value in the horizontal direction and the j-th input feature value in the vertical direction; A gating mechanism is used to allocate attention to different channel features. The gating mechanism consists of two 1×1 convolutional layers and a Relu activation function. First, the two first-order statistics obtained in the horizontal and vertical directions are concatenated in the channel dimension, and then a shared 1×1 convolution is performed to obtain the intermediate feature X1. Among them, Relu represents the activation function, W1 represents the convolution layer weight of the gating mechanism, and * represents the convolution operation of the gating mechanism; Split the intermediate feature X1 in the spatial dimension to obtain two identical tensors Then two 1×1 convolutional layers are used to obtain the horizontal attention att h and vertical attention att w ; Among them, Sigmoid represents the activation function, W h Represents the horizontal attention att h The weight of the upper convolutional layer, W w Represents vertical attention att w The weights of the upper convolutional layer; Finally, the horizontal attention att h and vertical attention att w Acting on the input feature X, the output feature is obtained 6. The light field image spatial super-resolution method based on multi-feature fusion according to claim 5, characterized in that: The step S4 comprises: First, the output features Reshape from macro-pixel mode to sub-aperture mode; Then use a 1×1 convolution layer to reshape the output features The depth is increased to 2αC, where α represents the upgrade factor and C represents the number of channels; Then, the output features with increased depth are transformed through a two-dimensional pixel transformation layer Upsampling to αAH z ×αAW z Resolution, where H z represents the height of the reshaped sub-aperture image, W z Represents the width of the sub-aperture image after reshaping; Finally, a 1×1 convolutional layer is used to compress the channels to obtain a super-resolution sub-aperture image.

Images