Light field image spatial super-resolution reconstruction method and system based on visual attention
By dividing the light field image into positive and negative sample blocks and using a visual attention mechanism for super-resolution reconstruction, combined with space-frequency loss function optimization training, the problem of wasted computational resources in light field image super-resolution reconstruction is solved, achieving efficient and clear image reconstruction results.
Patent Information
- Application Number
- CN202511810372.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-03
- Publication Date
- 2026-03-06
AI Technical Summary
Existing light field image super-resolution reconstruction methods consume large amounts of computational resources and have poor performance, which limits their application in mobile devices and embedded systems. Furthermore, existing methods fail to effectively utilize the selective attention mechanism of the human visual system.
The light field image is divided into positive and negative sample image blocks. A visual attention mechanism is used to perform super-resolution reconstruction through deep feature extraction and feature stitching. The training process is optimized by combining the spatial-frequency combination loss function, and the calculation is focused on key regions.
It significantly reduces computing resource consumption, improves image detail retention and visual effects, and supports high-quality display on resource-constrained devices.
Smart Images
Figure CN121616459A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and specifically to a method and system for spatial super-resolution reconstruction of light field images based on visual attention. Background Technology
[0002] Light field images, capable of recording the intensity and direction of light, provide users with immersive visual experiences such as multi-viewpoints and depth perception, demonstrating enormous application potential in fields such as virtual reality (VR), augmented reality (AR), autonomous driving, and medical imaging. However, limited by hardware devices and acquisition technologies, raw light field images often suffer from low spatial resolution, severely affecting the clarity of image details and the realism of the scene. Therefore, spatial super-resolution reconstruction technology for light field images has become a research hotspot. Deep learning-based spatial super-resolution reconstruction methods for light field images significantly improve the spatial resolution of light field images. These methods construct complex neural network models to learn the mapping relationship between low-resolution and high-resolution light field images, achieving the conversion from low-quality images to high-quality images and greatly improving the viewer's visual experience.
[0003] However, current mainstream methods generally adopt a strategy of uniformly processing the entire light field image. While this "one-size-fits-all" approach can achieve global super-resolution reconstruction, it has revealed many drawbacks in practical applications. On the one hand, because it requires full-scene rendering of all regions of the image, it places extremely high demands on the hardware performance of the device, resulting in huge consumption of hardware computing resources and a significant increase in operating costs. On the other hand, in most scenarios, the human visual system has a selective attention mechanism, only paying more attention to local areas of interest. However, existing methods also perform high-precision reconstruction on a large number of non-annotated regions, resulting in a serious waste of computing resources and limiting the widespread application of light field image super-resolution technology in resource-constrained scenarios such as mobile devices and embedded systems. Summary of the Invention
[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a light field image space super-resolution reconstruction method and system based on visual attention, which solves the problems of high computational resource consumption and poor performance in image super-resolution reconstruction.
[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: A method for super-resolution reconstruction of light field image space based on visual attention is provided, which includes the following steps: The target light field image is divided into several image blocks, and the image blocks are further divided into positive sample image blocks and negative sample image blocks based on their contribution to the reconstruction task. The initial features of the positive sample image patch are obtained by convolution, and the deep features corresponding to the initial features of the positive sample image patch are obtained by a deep feature extractor. The deep features are then added to the corresponding initial features pixel by pixel to obtain the deep features of the positive sample image patch. Shallow features of negative sample image patches are obtained by using convolution. The deep features of the positive sample image patch are concatenated with the shallow features of the negative sample image patch to obtain the feature image. By upsampling the feature image, the super-resolution reconstructed light field image space is obtained.
[0006] Furthermore, specific methods for dividing the target light field image into several image blocks include: The target light field image is divided into The image consists of 16 image blocks, where H and W are the spatial dimensions of the target light field image.
[0007] Furthermore, specific methods for dividing image patches into positive and negative sample image patches based on their contribution to the reconstruction task include: Image blocks with a non-black area ratio greater than 10% are considered to have a high contribution to the reconstruction task and are recorded as positive sample image blocks; image blocks with a non-black area ratio less than or equal to 10% are recorded as negative sample image blocks.
[0008] Furthermore, the convolution kernel size used to obtain the preliminary features of positive sample image patches is 3×3, and the corresponding expression is:
[0009] in Preliminary features of positive sample image patches; These are positive sample image patches; This indicates a convolution with a kernel size of 3×3.
[0010] Furthermore, the depth feature extractor includes a spatial feature extraction module, an angular feature extraction module, a pixel-by-pixel summation module, and a hybrid feature extraction module; the inputs to the spatial feature extraction module and the angular feature extraction module are both inputs to the depth feature extractor; the outputs of the spatial feature extraction module and the angular feature extraction module are both inputs to the pixel-by-pixel summation module; the output of the pixel-by-pixel summation module is the input to the hybrid feature extraction module; the output of the hybrid feature extraction module is the output of the depth feature extractor; the corresponding expression is:
[0011] in This represents the output of the deep feature extractor; Preliminary features of positive sample image patches; This represents the spatial feature extraction module; This represents the angle feature extraction module; This indicates the hybrid feature extraction module; This is the input for the pixel-by-pixel addition module.
[0012] Furthermore, the hybrid feature extraction module includes a spatial attention unit, a frequency attention unit, and a pixel-by-pixel summation unit; wherein the inputs of the spatial attention unit and the frequency attention unit are both inputs to the hybrid feature extraction module; the outputs of the spatial attention unit and the frequency attention unit are both inputs to the pixel-by-pixel summation unit; and the output of the pixel-by-pixel summation unit is the output of the hybrid feature extraction module; the corresponding expression is:
[0013] in This is the output of the frequency attention unit; This is the output of the spatial attention unit.
[0014] Furthermore, the spatial attention unit comprises a Linear layer, a GELU layer, a Conv1×1 layer, a Conv3×3 layer, and a pixel-wise multiplication layer connected in sequence; wherein the input of the Linear layer is the input of the spatial attention unit; the other input of the pixel-wise multiplication layer is the output of the Linear layer; the output of the pixel-wise multiplication layer is the output of the spatial attention unit; the corresponding expression is:
[0015] in Indicates a Linear layer; Indicates the GELU layer; Indicates Conv1×1 layer; This indicates a convolution with a kernel size of 3×3, i.e., a Conv3×3 layer.
[0016] Furthermore, the expression for the frequency attention unit is:
[0017] in and All are feature matrices. , .
[0018] A light field image spatial super-resolution reconstruction system based on visual attention is provided, comprising: The image preprocessing module is used to divide the target light field image into several image blocks, and to divide the image blocks into positive sample image blocks and negative sample image blocks based on their contribution to the reconstruction task. The feature acquisition module is used to obtain the preliminary features of positive sample image blocks by using convolution, and to obtain the deep features corresponding to the preliminary features of positive sample image blocks by using a deep feature extractor. The deep features are then added to the corresponding preliminary features pixel by pixel to obtain the deep features of the positive sample image blocks. The module also uses convolution to obtain the shallow features of negative sample image blocks. The super-resolution reconstruction module is used to stitch together the deep features of the positive sample image patch with the shallow features of the negative sample image patch to obtain a feature image; by upsampling the feature image, the light field image after spatial super-resolution reconstruction is obtained.
[0019] Furthermore, the feature acquisition module is obtained through pre-training, and the loss function during the pre-training process is:
[0020] in The loss function used in the pre-training process of the feature acquisition module; For L1 loss term, , The light field image obtained from super-resolution reconstruction. The label is the corresponding high-resolution ground truth map; For frequency loss term, , This represents the standard Fast Fourier Transform, which is the process of converting the image to the frequency domain using the standard Fast Fourier Transform. These are the weight parameters; during pre-training, the parameters updated in the feature acquisition module include the convolutional layer weights and biases, and the activation function weights and biases.
[0021] The beneficial effects of this invention are as follows: Compared with traditional Bicubic interpolation algorithms and deep learning methods such as EDSR, RCAN, and LFSSR, this invention can better capture and preserve the detailed texture information of images, presenting a clearer and more vivid visual effect. Detail comparisons show that the image reconstructed by this invention is clearer and more natural, with sharper object edges, and the overall visual quality is significantly better than other algorithms. The advantages of this invention can also be observed in the STFgantry_Cards scene. Attached Figure Description
[0022] Figure 1 This is a flowchart illustrating the method. Figure 2 This is a schematic diagram of the system structure; Figure 3 This is a schematic diagram of the structure of a deep feature extractor; Figure 4 This is a schematic diagram of the hybrid feature extraction module; Figure 5 This is a schematic diagram of the spatial attention unit. Figure 6 This is a schematic diagram of the structure of a frequency attention unit; Figure 7 The image shown is a reconstruction result from an example. Detailed Implementation
[0023] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0024] like Figure 1 As shown, this light field image space super-resolution reconstruction method based on visual attention includes the following steps: S1. Divide the target light field image into several image blocks, and divide the image blocks into positive sample image blocks and negative sample image blocks based on their contribution to the reconstruction task. S2. Use convolution to obtain the preliminary features of the positive sample image block, and use a deep feature extractor to obtain the deep features corresponding to the preliminary features of the positive sample image block. Then, add the deep features to the corresponding preliminary features pixel by pixel to obtain the deep features of the positive sample image block. S3. Use convolution to obtain shallow features of negative sample image patches; S4. The deep features of the positive sample image patch are concatenated with the shallow features of the negative sample image patch to obtain the feature image. S5. By upsampling the feature image, the super-resolution reconstructed light field image space is obtained.
[0025] like Figure 2 As shown, corresponding to the method, this light field image space super-resolution reconstruction system based on visual attention includes: The image preprocessing module is used to divide the target light field image into several image blocks, and to divide the image blocks into positive sample image blocks and negative sample image blocks based on their contribution to the reconstruction task. The feature acquisition module is used to obtain the preliminary features of positive sample image blocks by using convolution, and to obtain the deep features corresponding to the preliminary features of positive sample image blocks by using a deep feature extractor. The deep features are then added to the corresponding preliminary features pixel by pixel to obtain the deep features of the positive sample image blocks. The module also uses convolution to obtain the shallow features of negative sample image blocks. The super-resolution reconstruction module is used to stitch together the deep features of the positive sample image patch with the shallow features of the negative sample image patch to obtain a feature image; by upsampling the feature image, the light field image after spatial super-resolution reconstruction is obtained.
[0026] In this embodiment, the low-resolution light field image is represented as ,in For low-resolution light field images, R is the scope, H and W are the spatial dimensions, and U and V represent the angular dimensions (angular coordinates). The number of positive and negative sample image patches (non-critical image patches) that contribute significantly to the reconstruction task is: (i.e., 16), the reconstructed image is represented as (i=2,3,4), where This is a high-resolution light field image, where R is the scope. and For spatial dimensions.
[0027] In this embodiment, the specific method for dividing image patches into positive sample image patches and negative sample image patches based on their contribution to the reconstruction task includes: Image patches with a non-black area ratio greater than 10% are considered to have a high contribution to the reconstruction task and are denoted as positive sample image patches. Image patches with a non-black area ratio of less than or equal to 10% are designated as negative sample image patches. .
[0028] For negative sample image patches These are features that are not easily noticed by the human eye and have little impact on the reconstruction effect. They can be extracted using only a 3×3 convolutional layer.
[0029] In this embodiment, the convolution kernel size used to obtain the preliminary features of positive sample image patches is 3×3, and the corresponding expression is:
[0030] in Preliminary features of positive sample image patches; These are positive sample image patches; This indicates a convolution with a kernel size of 3×3.
[0031] like Figure 3As shown, the Deep Feature Extractor includes a Spatial Feature Extractor, an Angle Feature Extractor, a Pixel-by-Pixel Addition Module, and a Hybrid Feature Extraction Module. The inputs to both the Spatial and Angle Feature Extraction Modules are inputs to the Deep Feature Extractor. The outputs of both modules are inputs to the Pixel-by-Pixel Addition Module. The output of the Pixel-by-Pixel Addition Module is the input to the Hybrid Feature Extraction Module. The output of the Hybrid Feature Extraction Module is the output of the Deep Feature Extractor. The corresponding expression is:
[0032] in This represents the output of the deep feature extractor; Preliminary features of positive sample image patches; This represents the spatial feature extraction module; This represents the angle feature extraction module; This indicates the hybrid feature extraction module; This is the input for the pixel-by-pixel addition module.
[0033] like Figure 4 As shown, the hybrid feature extraction module includes a spatial attention unit, a frequency attention unit, and a pixel-by-pixel summation unit; the inputs of the spatial attention unit and the frequency attention unit are also inputs to the hybrid feature extraction module; the outputs of the spatial attention unit and the frequency attention unit are also inputs to the pixel-by-pixel summation unit; the output of the pixel-by-pixel summation unit is the output of the hybrid feature extraction module; the corresponding expression is:
[0034] in This is the output of the frequency attention unit; This is the output of the spatial attention unit.
[0035] like Figure 5As shown, the spatial attention unit comprises a Linear layer, a GELU layer, a Conv1×1 layer, a Conv3×3 layer, and a pixel-wise multiplication layer connected in sequence; the input of the Linear layer is the input of the spatial attention unit; the other input of the pixel-wise multiplication layer is the output of the Linear layer; the output of the pixel-wise multiplication layer is the output of the spatial attention unit; the corresponding expression is:
[0036] in Indicates a Linear layer; Indicates the GELU layer; Indicates Conv1×1 layer; This indicates a convolution with a kernel size of 3×3, i.e., a Conv3×3 layer.
[0037] like Figure 6 As shown, the frequency attention unit mainly consists of two feature matrices, and the expression for the frequency attention unit is:
[0038] in and All are feature matrices. , .
[0039] In this embodiment, the convolutional structure of the spatial feature extraction module, for which parameters are not explicitly defined, has a kernel size of 3×3, a stride of 1, and padding of U. The convolutional structure of the angle feature extraction module, for which parameters are not explicitly defined, has a kernel size of... The step size is U, and the padding size is 1.
[0040] In the specific implementation process, the feature acquisition module is obtained through pre-training, and the loss function during the pre-training process is:
[0041] in The loss function used in the pre-training process of the feature acquisition module; For L1 loss term, , The light field image obtained from super-resolution reconstruction. The label is the corresponding high-resolution ground truth map; For frequency loss term, , This represents the standard Fast Fourier Transform, which is the process of converting the image to the frequency domain using the standard Fast Fourier Transform. These are the weight parameters.
[0042] Traditional L1 loss functions focus on measuring the difference between super-resolution images and ground truth images in pixel space, but ignore the key high-frequency information contained in the image in the frequency domain. Therefore, this embodiment proposes a frequency loss function based on the L1 loss function to transform the super-resolution image and ground truth image to the frequency domain through standard fast Fourier transform. On this basis, a frequency loss function is constructed to calculate the image difference in the frequency domain, which can more accurately capture the high-frequency components such as image details and textures, thereby effectively improving the quality of the reconstructed image.
[0043] This embodiment ultimately yields the loss function. This loss function is used to adjust all trainable parameters θ in the system, specifically including the weights and biases of all convolutional layers and activation functions in the feature extraction module, and is continuously updated to the optimal solution as the network trains. This loss function combines information from the pixel space and frequency domain, and is applied during training. Larger weights are gradually reduced in the later stages of training. The weights are adjusted so that the network can finely adjust the overall structure and details of the image based on existing high-frequency features until the loss function converges. This dynamic weight adjustment strategy enables the network to quickly capture key high-frequency information during training and gradually optimize the overall image reconstruction effect, achieving fast and high-quality light field image space super-resolution reconstruction.
[0044] In light field image super-resolution reconstruction, original RGB images (PNG, JPG formats) suffer from large storage size and low training retrieval efficiency due to their high-dimensional spatial and angular information. To address this, this embodiment converts images to MAT format, which efficiently stores multi-dimensional data in binary form, significantly compressing data volume and improving loading speed. For each viewpoint image in each scene, a corresponding saliency region map needs to be generated. The LabelMe image annotation tool is used to generate black-and-white binary saliency region labels, marking visually important regions as non-black and other regions as black. This provides the system with a clear learning objective, enabling it to focus on key regions for super-resolution reconstruction. The initial processing of the training dataset is complete. To achieve better training results, data augmentation can be performed, primarily using a combination of methods such as cropping, flipping, and rotating images in the dataset. This increases data diversity, optimizes dataset quality, and effectively enhances the network's learning ability and robustness.
[0045] In one embodiment of the present invention, such as Figure 7As shown, we reconstructed intuitive images to demonstrate the results. The experimental results were compared with data from the HCI_new and STFgantry_Cards datasets. From the experimental results, we can clearly see that this invention achieves excellent reconstruction results in the following two scenarios. In the Bedroom / HCI_new scenario, compared with traditional Bicubic interpolation algorithms and deep learning methods such as EDSR, RCAN, and LFSSR, this invention can better capture and preserve the detailed texture information of the image, presenting a clearer and more vivid visual effect. Detail comparisons show that the image reconstructed by this invention is clearer and more natural, with sharper object edges, and the overall visual quality is significantly better than other algorithms. The advantages of this invention can also be observed in the STFgantry_Cards scenario. Compared with other algorithms, this invention can better preserve the detailed texture information in the image, resulting in a superior reconstruction effect.
[0046] Furthermore, as shown in Table 1, we conducted quantitative tests on a wider range of datasets. Comparing the quantitative metrics PSNR and SSIM, our invention achieved significant performance improvements on various dataset types, including HCI_new, INRIA, and STFgantry. Specifically, on the HCI_new dataset, our invention achieved a PSNR of 31.56 and an SSIM of 0.923, far exceeding other algorithms. This demonstrates that our invention effectively improves reconstruction accuracy and significantly enhances the viewing experience.
[0047] Table 1: Experimental Data Results of PSNR / SSIM
[0048] In summary, this invention effectively utilizes computational resources to focus on key areas, improving reconstruction accuracy and efficiency, and supporting excellent display effects on light field devices. It proposes a space-frequency combined loss function based on the L1Loss function, capturing high-frequency details through Fast Fourier Transform to enhance reconstruction quality. By optimizing computational resource allocation by focusing on key areas, it balances accuracy and efficiency, making it suitable for high-quality displays on light field devices.
Claims
1. A method for light field image spatial super-resolution reconstruction based on visual attention, characterized in that, The method comprises the following steps: dividing the target light field image into a plurality of image blocks, and dividing the image blocks into positive sample image blocks and negative sample image blocks based on contribution ability to a reconstruction task; obtaining preliminary features of the positive sample image blocks in a convolution manner, and obtaining deep features corresponding to the preliminary features of the positive sample image blocks through a deep feature extractor, and pixel-by-pixel adding the deep features and the corresponding preliminary features to obtain deep features of the positive sample image blocks; obtaining shallow features of the negative sample image blocks in a convolution manner; splicing the deep features of the positive sample image blocks and the shallow features of the negative sample image blocks to obtain a feature image; obtaining a light field image space after super-resolution reconstruction by upsampling the feature image.
2. The visual attention based light field image spatial super-resolution reconstruction method of claim 1, wherein, The specific method for dividing the target light field image into a plurality of image blocks comprises: The target light field image is divided into 16 image patches, where H and W are the spatial dimensions of the target light field image.
3. The visual attention based light field image spatial super-resolution reconstruction method of claim 1, wherein, The specific method for dividing the image blocks into positive sample image blocks and negative sample image blocks based on contribution ability to a reconstruction task comprises: image blocks with a non-black area ratio greater than 10% are regarded as image blocks with high contribution ability to the reconstruction task, and are recorded as positive sample image blocks; and image blocks with a non-black area ratio less than or equal to 10% are recorded as negative sample image blocks.
4. The visual attention based light field image spatial super-resolution reconstruction method of claim 1, wherein, The convolution kernel size of the convolution used for obtaining the preliminary features of the positive sample image blocks is 3x3, and the corresponding expression is: wherein is a preliminary feature of the positive sample image block; is a positive sample image block; denotes a convolution with a kernel size of 3x3.
5. The visual attention based light field image spatial super-resolution reconstruction method of claim 1, wherein, The deep feature extractor comprises a spatial feature extraction module, an angle feature extraction module, a pixel-by-pixel addition module and a mixed feature extraction module; the input of the spatial feature extraction module and the input of the angle feature extraction module are both the input of the deep feature extractor; the output of the spatial feature extraction module and the output of the angle feature extraction module are both the input of the pixel-by-pixel addition module; the output of the pixel-by-pixel addition module is the input of the mixed feature extraction module; the output of the mixed feature extraction module is the output of the deep feature extractor; and the corresponding expression is: wherein represents an output of the deep feature extractor; is a preliminary feature of the positive sample image block; represents a spatial feature extraction module; represents an angle feature extraction module; represents a hybrid feature extraction module; is an input of the pixel-wise addition module.
6. The visual attention based light field image spatial super-resolution reconstruction method of claim 5, wherein, The mixed feature extraction module comprises a spatial attention unit, a frequency attention unit and a pixel-by-pixel addition unit; the input of the spatial attention unit and the input of the frequency attention unit are both the input of the mixed feature extraction module; the output of the spatial attention unit and the output of the frequency attention unit are both the input of the pixel-by-pixel addition unit; the output of the pixel-by-pixel addition unit is the output of the mixed feature extraction module; and the corresponding expression is: wherein is the output of the frequency attention unit; is the output of the spatial attention unit.
7. The visual attention based light field image spatial super-resolution reconstruction method of claim 6, wherein, The spatial attention unit comprises sequentially connected Linear layer, GELU layer, Conv1x1 layer, Conv3x3 layer and pixel-by-pixel multiplication layer; the input of the Linear layer is the input of the spatial attention unit; the other input of the pixel-by-pixel multiplication layer is the output of the Linear layer; the output of the pixel-by-pixel multiplication layer is the output of the spatial attention unit; and the corresponding expression is: wherein represents a Linear layer; represents a GELU layer; represents a Conv1x1 layer; represents a convolution with a kernel size of 3x3, i.e. a Conv3x3 layer.
8. The visual attention based light field image spatial super-resolution reconstruction method of claim 6, wherein, The expression of the frequency attention unit is: wherein and are both feature matrices, , .
9. A system for visual attention based light field image spatial super-resolution reconstruction according to any of claims 1 to 8, characterized in that, The method comprises the following steps: an image preprocessing module, configured to divide the target light field image into a plurality of image blocks, and divide the image blocks into positive sample image blocks and negative sample image blocks based on contribution ability to a reconstruction task; The feature acquisition module is configured to acquire a preliminary feature of the positive sample image block in a convolution manner, acquire a deep feature corresponding to the preliminary feature of the positive sample image block through a deep feature extractor, and add the deep feature and the corresponding preliminary feature pixel by pixel to obtain the deep feature of the positive sample image block; and acquire a shallow feature of the negative sample image block in a convolution manner. The super-resolution reconstruction module is configured to splice the deep feature of the positive sample image block and the shallow feature of the negative sample image block to obtain a feature image, and obtain a spatial super-resolution reconstructed light field image by upsampling the feature image.
10. The system of claim 9, wherein, The feature acquisition module is obtained through pre-training, and a loss function in the pre-training process is: wherein is a loss function in the pre-training process of the feature acquisition module; is an L1 loss term, , is a light field image obtained by super-resolution reconstruction, is a label, i.e., a corresponding high-resolution ground truth image; is a frequency loss term, , represents a standard fast Fourier transform, i.e., the frequency of an image is converted into a frequency domain by a standard fast Fourier transform; is a weight parameter; in the pre-training process, the parameters of the feature acquisition module to be updated include convolution layer weights and biases, and activation function weights and biases.