Image super-resolution method for capturing long-distance and global information in high-dimensional information space
By constructing the SSM-SWTrans dual-path hybrid network, differential feature learning and fusion are performed on three types of 2D slices after light field decoupling. This solves the problems of low efficiency in long-distance information modeling and imbalance between local and global features in light field imaging, and achieves efficient and high-precision light field reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-24
- Publication Date
- 2026-03-27
AI Technical Summary
Existing light field imaging technologies suffer from problems such as low efficiency in long-distance information modeling, imbalance between local and global features, and insufficient module coordination, which limit reconstruction accuracy.
A dual-path hybrid network, SSM-SWTrans, is constructed to learn differentiated features of three types of 2D slices after light field decoupling. By combining dual attention fusion and sub-pixel convolution reconstruction, a balance between high computational efficiency and high reconstruction accuracy is achieved.
It improves computational efficiency, enhances the PSNR and SSIM of reconstructed images, significantly improves the continuity of edges and textures in visual effects, and reduces the development cost of multi-scene applications.
Smart Images
Figure CN121746181A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, in particular to an image super-resolution method for long-distance and global information capture in high-dimensional information space. BACKGROUND
[0002] Light field imaging forms a 4D data structure by simultaneously capturing the intensity and direction information of light rays, providing core support for three-dimensional reconstruction, depth estimation, dynamic focusing, and other tasks, and has wide application potential in automatic driving, medical imaging, virtual reality, and other fields. However, light field imaging has an inherent trade-off between spatial and angular resolution. The multiplexing of sensor pixels results in a significant decrease in the spatial resolution of single-view sub-aperture images, which is a key bottleneck for technology landing. The development of light field super-resolution technology has gone through three stages. Early traditional optimization methods, represented by LFBM5D, divide the image blocks and calculate the non-local similarity to mine high-frequency information. Although they are better than basic methods such as bicubic interpolation, they rely on artificially designed feature rules and have limited performance improvement. Mid-term deep learning methods extend single-image super-resolution networks (such as RCAN) to the light field field. Works such as LF-InterNet and DistgSSR use sub-aperture and macro-pixel 2D slice characteristics to preliminarily fuse spatial-angular information through customized convolution kernels. However, the local bias of convolution makes it difficult to capture long-distance dependencies. Recent methods based on Transformer (such as LFT and EPIT) further improve performance by modeling global associations through self-attention mechanisms. However, the quadratic computational complexity of self-attention and the problem of redundant feature interference are prominent, making it difficult to adapt to the high-dimensional data characteristics of 4D light fields. However, the existing technology still has obvious defects. First, the efficiency of long-distance information modeling is low. CNNs cannot capture the depth correlation corresponding to the slope in EPI due to the limitation of the receptive field. Transformers are difficult to handle full-size light field data due to the explosion of computational complexity. Second, the feature utilization is not accurate. No differential modeling strategy is designed for the spatial texture of SAI, the angle-depth correlation of MacPI and EPI, leading to an imbalance between local details and global structure. Third, the module collaboration is insufficient. A single path cannot balance long-distance dependencies and local details, and feature fusion lacks targeted weight allocation, which restricts the reconstruction accuracy.
[0003] Therefore, there is an urgent need in the art for an image super-resolution method for long-distance and global information capture in high-dimensional information space to solve the above problems. SUMMARY
[0004] The application provides an image super-resolution method for long-distance and global information capture in high-dimensional information space, aiming to solve the problems of low modeling efficiency of long-distance information in light field super-resolution and imbalance between local and global features, and realizes the unification of high calculation efficiency and high reconstruction accuracy by constructing an SSM-SWTrans double-path hybrid network, differentiating feature learning on three types of 2D slices after light field decoupling, combining double attention fusion and sub-pixel convolution reconstruction, and breaking through the space-angle resolution balance bottleneck of light field.
[0005] The application provides an image super-resolution method for long-distance and global information capture in high-dimensional information space, comprising the following steps: Step one, decoupling processing of four-dimensional light field data to obtain three types of two-dimensional slices of sub-aperture images SAI, macro-pixel images MacPI and epi-plane images EPI; Step two, constructing a hybrid network structure, wherein the hybrid network structure comprises an initial feature extraction module IFE, a space-angle feature learning module SAFL, a light field structure feature learning module LSFL and a high-resolution reconstruction module HLFR, and SSM-SWTransBlock modules are stacked in the SAFL and the LSFL; Step three, inputting the three types of two-dimensional slices into the hybrid network structure, modeling long-distance information and perceiving local information on different two-dimensional slices through the SSM-SWTransBlock modules, and completing multi-dimensional feature fusion; Step four, using sub-pixel convolution PixSF in the high-resolution reconstruction module HLFR for joint upsampling to output a high-resolution light field image.
[0006] Compared with the prior art, the application has the following advantages: 1. The application adopts a selective state space model (SSM) to construct a linear complexity path, covers the full-angle-space correlation of light field two-dimensional slices through ESS2D four-way scanning, can improve the calculation efficiency when processing EPI long-distance depth information compared with the quadratic complexity of Transformer, and retains key structure features.
[0007] 2. The application designs a differential processing strategy for the characteristics of three types of two-dimensional slices, the sliding window attention of the SWTrans path accurately mines the spatial texture details of SAI, the SSM path efficiently captures the angle-depth correlation of MacPI and EPI, solves the problem of "local and global information being lost" in traditional methods, and improves the peak signal-to-noise ratio (PSNR) of the reconstructed image.
[0008] 3. The application dynamically allocates weights through the channel-space dual attention mechanism, strengthens the key features of texture details and depth structure, suppresses redundant information interference, and improves the structural similarity (SSIM) compared with the single feature fusion method, significantly improving the continuity of edges and textures in visual effects.
[0009] 4. The core module topology of the application is fixed, and only the output channel of the reconstruction module and the loss function need to be adjusted to adapt to tasks such as denoising and artifact removal, without the need for a reconstruction network, reducing the development cost of multi-scene applications and improving the flexibility of technology landing.
[0010] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the application. The objects and other advantages of the present application can be achieved and obtained by the structure specifically pointed out in the written description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS
[0011] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation of the present application. In the drawings: Figure 1 A flowchart of an image super-resolution method for high-dimensional information space long-distance and global information capture according to the present application is provided. Figure 2 A process diagram for expanding 4D light field information into sub-aperture image 2D data is provided. Figure 3 A process diagram for expanding 4D light field image into macro-pixel image 2D data is provided. Figure 4 A process diagram for expanding 4D light field image into 2D polar plane image data is provided. Figure 5 A network structure diagram of the super-resolution algorithm according to the present application is provided. Figure 6 A network structure diagram of the core SSM-SWTrans module according to the present application is provided. DETAILED DESCRIPTION
[0012] The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0013] Example 1: Please refer to Figures 1-6The application provides an image super-resolution method for long-distance and global information capture in a high-dimensional information space, comprising the following steps: Step one: decoupling processing of four-dimensional (4D) light field data to obtain three types of two-dimensional (2D) slices, i.e., sub-aperture images SAI, macro-pixel images MacPI and extreme plane images EPI; Step two: constructing a hybrid network structure, wherein the hybrid network structure comprises an initial feature extraction module IFE, a space-angle feature learning module SAFL, a light field structure feature learning module LSFL and a high-resolution reconstruction module HLFR, and SSM-SWTransBlock modules are stacked in SAFL and LSFL; Step three: inputting the three types of 2D slices into the hybrid network structure, modeling long-distance information and perceiving local information of different 2D slices through SSM-SWTransBlock modules, and completing multi-dimensional feature fusion; Step four: performing joint up-sampling by using a sub-pixel convolution PixSF in the high-resolution reconstruction module HLFR to output a high-resolution light field image.
[0014] Specifically, the method is used for a light field image in the form of 4D (x, y, u, v) recording spatial position and angle direction information, and realizes super-resolution processing through a whole process of decoupling, modeling, fusion and reconstruction; wherein the decoupling process of 4D light field data corresponds to Figure 2 , and Figure 3 , and Figure 4 are shown in , and Figure 2 (Macro-pixel image decoupling) 4D light field data is defined as , and is decomposed into a macro-pixel image set , wherein is a single macro-pixel image, and the superscript K corresponds to a spatial coordinate index (the value range is 1 to , which indicates that each macro-pixel image uniquely corresponds to a spatial position (x, y) in the light field; each has a dimension of , that is, the macro-pixel image records the light information (which can be understood as "a full-angle light set of the same spatial point") of all angles (u, v) under the corresponding spatial position (x, y); that is, the full-angle (u, v) information is extracted under the fixed spatial position (x, y) to form a macro-pixel image with a dimension of ; , and Figure 3 (Sub-aperture image decoupling) 4D light field data is defined as , and is decomposed into a sub-aperture image set , wherein is a single sub-aperture image, and the superscript K corresponds to an angle coordinate index (the value range is 1 to , which indicates that each sub-aperture image uniquely corresponds to a set of angles (u, v) in the light field; each dimension of , that is, the sub-aperture image records the spatial pixel information (which can be understood as a "spatial image under a single view") of the entire scene under the corresponding angle (u, v); that is, the full spatial (x, y) information is extracted under the fixed angle (u, v) to form sub-aperture images of dimension; attached Figure 4 (EPI decoupling) 4D light field data is defined as , and is decoupled into a set of EPIs or , which indicates that the 4D light field can be decoupled into a set of EPIs, and is specifically divided into two typical cross-sectioning methods; the first method is to extract the data of the remaining angle dimension U and spatial dimension H by fixing a coordinate in the angle dimension V and a coordinate in the spatial dimension W, at this time, the dimension of a single EPI is , and the total number of the set is (that is, K is from 1 to ); the second method is to extract the data of the remaining angle dimension V and spatial dimension W by fixing a coordinate in the angle dimension U and a coordinate in the spatial dimension H, at this time, the dimension of a single EPI is , and the total number of the set is (that is, K is from 1 to ); that is, by fixing the spatial-angle cross-section (such as fixing V and W or U and H), the remaining dimension information is extracted to form the cross-sectioned EPIs; The overall topology of the mixed network structure corresponds to that shown in attached Figure 5 , the network sequentially passes through the initial feature extraction module IFE, the spatial-angle feature learning module SAFL, the light field structure feature learning module LSFL, and the high-resolution reconstruction module HLFR in the form of four cascades, and the "BasicSSMBlock" stacked inside SAFL and LSFL is replaced by the "SSM-SWTransBlock" module; multi-dimensional feature fusion is realized through the cooperation of the double paths inside the "SSM-SWTransBlock" module, and finally the upsampling is completed through the sub-pixel convolution (PixSF) in the high-resolution reconstruction module shown in attached Figure 6 , and the high-resolution (HR) light field image is directly output without relying on traditional upsampling methods such as bilinear or cubic interpolation.
[0015] In one embodiment, when decoupling the 4D light field data in step one, the sub-aperture image SAI reflecting spatial texture and short-range dependence, the macro-pixel image MacPI representing angle-depth correlation, and the polar plane image EPI reflecting long-distance angle information are extracted according to the differences in spatial position, angular direction and structural features of the 4D light field data.
[0016] Specifically, the decoupling of 4D light field data is based on its inherent spatial-angular resolution balance characteristics, and it is divided into three types of 2D slices to meet the expression needs of different dimensions of information. Sub-aperture image (SAI): corresponding appendix Figure 3 The decoupling logic, mathematically defined as For spatial dimension details, each SAI corresponds to a slice with a fixed (u,v) angle and varying (x,y) space in the 4D light field, which can intuitively reflect the spatial texture of the scene and the short-range pixel dependency. The unfolding process is achieved by using fixed angle coordinates. Extract spatial dimension (x, y) data to form a spatial image from a single perspective; Macropixel Image (MacPI): Corresponding Appendix Figure 2 The decoupling logic, mathematically defined as Using spatial position (x, y) as a fixed dimension, angle dimension (u, v) data is extracted. Each macro-pixel corresponds to the light information at different angles within the same spatial position, which can directly represent the relationship between angle and scene depth. The unfolding process involves using fixed spatial coordinates... Integrate angular dimension (u,v) data to form a macro-pixel array with dense angular information; Polar plane image (EPI): corresponding appendix Figure 4 The decoupling logic, mathematically defined as or The spatial-angle cross-slicing method is used, fixing one spatial dimension and one angular dimension, and extracting data from the other spatial dimension and the other angular dimension. The slope of straight lines in the image directly corresponds to the scene depth, and the pixel span along the angular dimension is large, which can reflect long-distance angular information correlation. The unfolding process involves fixing coordinates (such as...) Extract (y,v) dimension data to form polar plane slices that combine spatial and angular long-range information.
[0017] In one embodiment, the SSM-SWTransBlock module in step two includes an SSM path and a SWTrans path. After the input feature map is normalized by LayerNorm, it is fed into the two paths for parallel processing. The outputs of the two paths are concatenated in the channel dimension and then compressed to a preset dimension by a 1×1 convolution to complete feature fusion.
[0018] Specifically, the SSM-SWTransBlock module is the core functional module of the hybrid network, and its structure corresponds to the attached Figure 6 As shown in the SSM-SWTrans module in the middle, the input feature map is first normalized by the LayerNorm layer to eliminate the influence of the distribution difference of the feature values on subsequent modeling; the normalized feature map is simultaneously split into the SSM path and the SWTrans path: the SSM path is responsible for long-distance information modeling, and the SWTrans path is responsible for local information perception, and the two paths operate in parallel to ensure processing efficiency; after the two paths respectively output the feature maps, the two feature maps are spliced in the channel dimension, at which time the number of channels of the feature map is the sum of the number of output channels of the two paths (assuming that the number of output channels of a single path is C, then after splicing it is 2C); finally, the 2C-dimensional feature map after splicing is compressed in the channel by a 1x1 convolutional layer, the number of channels is restored to the preset C dimension, realizing information integration and dimension unification of the features of the two paths, completing the feature fusion process, and providing collaborative optimization feature input for subsequent modules.
[0019] In one embodiment, the SSM path adopts a selective state space model (VSSM / Mamba) to perform ESS2D four-way scanning on the input 2D slice, converting the 2D slice into a 1D sequence with a length of HxW, where H is the image height and W is the image width; and then completing global long-distance dependency modeling with linear complexity through a selective state space equation.
[0020] Specifically, the selective state space model adopted by the SSM path is a linear complexity model, the core of which is to realize efficient capture of long-distance information through a state space equation, avoiding the quadratic complexity problem of traditional Transformer self-attention. For a 2D slice feature map with an input size of HxWxC (H is the image height, W is the image width, and C is the number of channels), first, perform ESS2D (Efficient State Space 2D) four-way scanning, i.e., scan the 2D feature map in four directions of "up-down", "left-right", "top-left-bottom-right", and "top-right-bottom-left", respectively, and convert the scanning results of each direction into a 1D sequence with a length of HxW, ensuring that all spatial position correlations of the pixels in the 2D slice are covered; then input the obtained 1D sequence into the selective state space model, and complete modeling through a state space equation (where X is the input 1D sequence, and Y is the output feature sequence) to complete modeling, which dynamically retains key long-distance information through a selective gating mechanism while filtering redundant information, and the time complexity of the entire modeling process is linearly related to the sequence length (i.e., O(L), L=HxW), which can control the calculation cost while effectively modeling the global long-distance dependency of the light field image.
[0021] In one embodiment, the SWTrans path adopts a sliding window Transformer structure, and first performs WindowPartition processing on the normalized feature map, divides the feature map of HxW size into multiple non-overlapping kxk windows, where k is the window side length; within each window, the features are flattened into k2 tokens, the exchange of local information and global information within the window is realized through multi-head self-attention MSA, and the window offset step is set to k / / 2, ensuring that the adjacent window edge pixels obtain cross-window context information.
[0022] Specifically, the SWTrans path takes Swin-Transformer as the basic architecture, and for the normalized feature map with a size of HxWxC, first performs WindowPartition operation, divides the HxW feature map plane into multiple non-overlapping kxk windows according to a preset window side length k (such as k=4, k=8, etc.), and if H or W cannot be divided by k, the size is padded by zero to ensure that the size of each window is uniform; for each kxk window, the kxkxC features inside the window are flattened along the spatial dimension and converted into k 2 tokens with a dimension of C, so that the pixel information within the window participates in the subsequent operation in the form of a sequence; then the token sequence is input into a multi-head self-attention (MSA) module, the correlation weight between tokens is calculated through multiple parallel attention heads, the deep interaction of local information within the window is realized, and the exchange of local and global information is indirectly completed through global allocation of attention weights; to solve the problem of context missing of edge pixels caused by window partition, the window offset step is set to k / / 2 (i.e. half of the window side length), and in the process of adjacent window processing, the window is offset by k / / 2 pixels along the horizontal and vertical directions, so that the edge pixels of adjacent windows can be covered by multiple windows, thereby obtaining the context information across the windows and ensuring the continuity of local texture and structure.
[0023] In one embodiment, when different 2D slices are processed in step three, for the sub-aperture image SAI, the local window attention mechanism of the SWTrans path is used to mine spatial texture and short-range dependence; for the macro-pixel image MacPI and the epipolar plane image EPI, the multi-directional scanning of the SSM path is used to realize long-distance angle-depth correlation modeling.
[0024] Specifically, based on the differences in information characteristics of the three types of 2D slices, a divide-and-conquer processing strategy is adopted: For the sub-aperture image (SAI), the corresponding decoupling result Figure 3 is obtained The core value is the spatial texture details and short-range pixel dependence, such as the edge, texture, color transition and other local features in the scene, and the local window attention mechanism (MSA) of the SWTrans path can accurately capture the short-range correlation of pixels within the window, and the sliding window design ensures the continuity of local features, so for the SAI slice, after inputting it into the SSM-SWTransBlock module, it mainly relies on the SWTrans path for processing, and through the window attention calculation, the spatial texture details are mined, and the interference of long-distance redundant information on local features is suppressed; For the macro-pixel image (MacPI) and the epipolar plane image (EPI), the MacPI corresponds to the decoupling result of the attached Figure 2 The decoupling result of the attached The EPI corresponds to the decoupling result of the attached Figure 4 The decoupling result of the attached Or The core information of the two is the long-distance angle-depth correlation (the long-distance correlation of the full-angle light rays at the same spatial position in MacPI, and the long-distance pixel correlation in the cross-dimension in EPI), and the four-way scanning and linear complexity state space model of the SSM path can efficiently cover the long-distance pixel correlation, and avoid the explosion of calculation amount, so for the MacPI and EPI slices, after inputting them into the SSM-SWTransBlock module, they mainly rely on the SSM path for processing, the long-distance correlation of the angle and depth dimensions is covered through multi-directional scanning, and then the state space equation modeling is realized. The angle-depth information is accurately extracted.
[0025] In one embodiment, the multi-dimensional feature fusion in step three adopts a channel-space dual attention mechanism, which performs weight allocation and information integration on the global features output by the SSM path and the local features output by the SWTrans path, to obtain the cooperatively optimized light field features.
[0026] Specifically, the multi-dimensional feature fusion employs a channel-space dual attention mechanism to achieve deep integration of the global long-range features output by the SSM path and the local detail features output by the SWTrans path in the SSM-SWTransBlock module. In the channel dimension, the channel attention module calculates the importance weight of each channel, assigning higher weights to channels with strong angle-depth correlation in global features and channels with strong spatial texture correlation in local features to suppress interference from redundant channels. In the spatial dimension, the spatial attention module calculates the weight of each pixel position, assigning weights to pixel regions with significant long-range correlation in global features and pixel regions with rich texture details in local features. Pixel regions are assigned higher weights to ensure the preservation of key spatial information. The weight calculation of the dual attention mechanism is achieved through the "attention map generation - weight multiplication and addition" process. First, global average pooling and convolution operations are performed on the feature maps of the two paths to generate channel attention weight maps and spatial attention weight maps. Then, the weight maps are multiplied and added to the feature maps of the corresponding paths one channel at a time, one pixel at a time, to complete the weight allocation. Finally, the weighted global features and local features are concatenated in the channel dimension and then fused in dimension by 1×1 convolution to obtain the collaboratively optimized light field features that contain both global long-distance correlation and local detail information, providing comprehensive feature support for subsequent structure learning and reconstruction.
[0027] In one embodiment, the high-resolution reconstruction module HLFR in step four takes the collaboratively optimized light field features output in step three as input, first performs feature adjustment through a convolutional layer, and then uses sub-pixel convolution PixSF technology to jointly upsample the feature map, directly generating a high-resolution light field image of a preset resolution without intermediate interpolation.
[0028] Specifically, the structure of the High Resolution Reconstruction Module (HLFR) corresponds to the attached... Figure 6 The "High-Resolution Reconstruction Module" shown takes the co-optimized light field features (H×W×C) output from step three as input. Specifically, it first uses a 1×1 convolutional layer to adjust the channels of the input features, converting the number of feature channels to a preset "upsampling factor". 2 × number of output channels (e.g., if the target upsampling factor is 2, the number of channels is adjusted to 4×3, where 3 is the number of output channels for the RGB image), ensuring that subpixel convolution can directly generate an image at the target resolution; subsequently, subpixel convolution (PixSF) is used to perform upsampling operations. Subpixel convolution achieves resolution improvement by mapping the channel dimension information of the input feature map to the spatial dimension. For example, for an image of size H×W×(r 2feature maps of (rH) x (rW) x Cout, the whole process has no intermediate interpolation step, avoiding the detail blur caused by traditional interpolation (such as bilinear, cubic interpolation); finally, the high-resolution feature maps after remodeling are optimized for details through a 3x3 convolution layer, eliminating the checkerboard artifacts that may be introduced by sub-pixel convolution, and directly outputting a high-resolution light field image of a preset resolution (rH x rW), and the angle dimension (U x V) of the output image is consistent with the input 4D light field (such as the attached Figures 2-4 defined ) to ensure the integrity of the light field.
[0029] In an embodiment, the hybrid network structure can adapt to image super-resolution, image denoising, compression artifact removal and various image restoration tasks, and only needs to adjust the output channel number of the high-resolution reconstruction module HLFR and the loss function for different tasks, without changing the overall topology of the network and the core parameters of the SSM-SWTransBlock module.
[0030] Specifically, the versatility of the hybrid network structure comes from its hierarchical design of "feature learning-task adaptation", that is, the core modules of the network (initial feature extraction module IFE, spatial-angle feature learning module SAFL, light field structure feature learning module LSFL) and SSM-SWTransBlock module, which are responsible for extracting the spatial-angle-structure features common in the light field image. These features have commonality in image super-resolution, denoising, and compression artifact removal tasks (such as spatial texture, angle correlation, and structure consistency), so there is no need to adjust the topology and parameters of the core modules (such as the four-way scanning direction of the SSM path, the window side length k of the SWTrans path, the number of attention heads, etc.) for different tasks; task adaptation is only realized through the high-resolution reconstruction module (HLFR): for image super-resolution tasks, the output channel number of HLFR is set to the color channel number of the light field image (such as 3 for RGB images), and the loss function adopts L1 loss or perceptual loss to optimize the pixel difference and visual perceptual consistency between high and low resolution images; for image denoising tasks, the output channel number of HLFR is also set to the color channel number, and the loss function adopts L2 loss or total variation loss adapted to Gaussian noise to suppress noise while preserving details; for compression artifact removal tasks, the output channel number of HLFR is set to the color channel number, and the loss function adopts a combination of adversarial loss and perceptual loss for compression artifacts to eliminate artifacts such as blocking effects; by adjusting only the output channel number and loss function of HLFR as described above, the hybrid network can adapt to various image restoration tasks without reconstruction, improving the practicality and adaptability of the method.
[0031] In one embodiment, the initial feature extraction module IFE in step two adopts a 3x3 convolution layer combined with a ReLU activation function structure to perform preliminary feature extraction on the input 2D slice, and convert the original image data into a feature map with a preset number of channels, thereby providing basic feature input for the subsequent spatial-angle feature learning module SAFL and light field structure feature learning module LSFL Specifically, the structure of the initial feature extraction module (IFE) corresponds to the structure shown in FIG. 4 Figure 5 In the "initial feature extraction module" part, the core function is to convert the input original 2D slice (such as SAI, MacPI, EPI, with a size of HxWxC_in, C_in is the input channel number, such as 3 for an RGB image) into a feature map suitable for subsequent deep learning. This module adopts a combined architecture of "3x3 convolution layer + ReLU activation function": the number of convolution kernels of the 3x3 convolution layer is set to a preset number of channels C (such as C=64, C=128, etc.), and the HxWxC_in original image data (such as FIG. 4 Figure 3 , FIG. 4 Figure 2 , FIG. 4 Figure 4 or ) is mapped to a feature map of HxWxC through convolution operation, and the 3x3 convolution kernel can control the number of calculation parameters while ensuring the coverage of the local pixel correlation of the receptive field; the ReLU activation function introduces feature nonlinear expression ability through the nonlinear transformation of f(x)=max(0,x), suppresses the propagation of invalid features, and enhances the feature discrimination; after IFE processing, the original image data is converted into a basic feature map with unified dimensions and strong expression ability, which is simultaneously input into the spatial-angle feature learning module (SAFL) and the light field structure feature learning module (LSFL), thereby providing consistent feature input basis for the two modules and ensuring the effectiveness of subsequent long-distance modeling and local perception.
[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An image super-resolution method for capturing long-distance and global information in a high-dimensional information space, characterized in that, Includes the following steps: Step 1: Decouple the four-dimensional light field data to obtain three types of two-dimensional slices: sub-aperture image (SAI), macropixel image (MacPI), and epipolar plane image (EPI). Step 2: Construct a hybrid network structure, which includes an initial feature extraction module IFE, a spatial-angle feature learning module SAFL, an optical field structure feature learning module LSFL, and a high-resolution reconstruction module HLFR. The SAFL and LSFL are stacked with an SSM-SWTransBlock module. Step 3: Input the three types of two-dimensional slices into the hybrid network structure, and use the SSM-SWTransBlock module to perform long-distance information modeling and local information perception on different two-dimensional slices respectively, and complete multi-dimensional feature fusion. Step four: Joint upsampling is performed using the sub-pixel convolution PixSF in the high-resolution reconstruction module HLFR to output a high-resolution light field image.
2. The super-resolution method for capturing light field images with long-distance and global information in high-dimensional information space according to claim 1, characterized in that, In step one, when decoupling the four-dimensional light field data, the sub-aperture image SAI reflecting spatial texture and short-range dependence, the macro-pixel image MacPI representing angle-depth correlation, and the polar plane image EPI reflecting long-distance angle information are extracted according to the differences in spatial position, angular direction and structural features of the four-dimensional light field data.
3. The super-resolution method for capturing light field images with long-distance and global information in high-dimensional information space according to claim 1, characterized in that, In step two, the SSM-SWTransBlock module includes the SSM path and the SWTrans path. After the input feature map is normalized by LayerNorm, it is fed into the two paths for parallel processing. The outputs of the two paths are concatenated in the channel dimension and then compressed to the preset dimension by 1×1 convolution to complete feature fusion.
4. The super-resolution method for capturing light field images with long-distance and global information in high-dimensional information space according to claim 3, characterized in that, The SSM path employs a selective state-space model, performing an ESS2D four-way scan on the input 2D slice to convert the 2D slice into a 1D sequence of length H×W, where H is the image height and W is the image width; then, the selective state-space equation is used to complete the global long-distance dependency modeling with linear complexity.
5. The super-resolution method for capturing light field images with long-distance and global information in high-dimensional information space according to claim 3, characterized in that, The SWTrans path employs a sliding window Transformer structure. First, the normalized feature map undergoes WindowPartition processing, dividing the H×W feature map into multiple non-overlapping k×k windows, where k is the window side length. Within each window, the feature map is flattened into k... 2 Each token is used to exchange local and global information within a window through multi-head self-attention (MSA), and the window offset step size is set to k / / 2 to ensure that adjacent window edge pixels obtain cross-window context information.
6. The super-resolution method for capturing light field images with long-distance and global information in high-dimensional information space according to claim 1, characterized in that, In step three, when processing different two-dimensional slices, for the sub-aperture image SAI, the spatial texture and short-range dependencies are mined through the local window attention mechanism of the SWTrans path; for the macro-pixel image MacPI and the polar plane image EPI, long-distance angle-depth correlation modeling is achieved through multi-directional scanning of the SSM path.
7. The super-resolution method for capturing light field images with long-distance and global information in high-dimensional information space according to claim 1, characterized in that, In step three, the multi-dimensional feature fusion adopts a channel-space dual attention mechanism to perform weight allocation and information integration on the global features output by the SSM path and the local features output by the SWTrans path, so as to obtain the co-optimized light field features.
8. The super-resolution method for capturing light field images with long-distance and global information in high-dimensional information space according to claim 1, characterized in that, In step four, the high-resolution reconstruction module HLFR takes the collaboratively optimized light field features output from step three as input. It first performs feature adjustment through a convolutional layer, and then uses sub-pixel convolution PixSF technology to jointly upsample the feature map, directly generating a high-resolution light field image with a preset resolution without intermediate interpolation.
9. The super-resolution method for capturing light field images with long-distance and global information in high-dimensional information space according to claim 1, characterized in that, The hybrid network structure can be adapted to various image restoration tasks such as image super-resolution, image denoising, and compression artifact removal. For different tasks, only the number of output channels and loss function of the high-resolution reconstruction module HLFR need to be adjusted, without changing the overall network topology and the core parameters of the SSM-SWTransBlock module.
10. The super-resolution method for capturing light field images with long-distance and global information in high-dimensional information space according to claim 1, characterized in that, In step two, the initial feature extraction module IFE uses a combination of 3×3 convolutional layers and ReLU activation function to perform preliminary feature extraction on the input two-dimensional slices, converting the original image data into feature maps with a preset number of channels, providing basic feature input for the subsequent spatial-angle feature learning module SAFL and light field structure feature learning module LSFL.