A Stereo Image Super-Resolution Method Based on Progressive Feature Extraction via Frequency Separation

Abstract

This invention discloses a stereo image super-resolution method based on progressive feature extraction using frequency separation, specifically relating to the field of image super-resolution technology. The proposed method includes the following steps: preprocessing the low-resolution image; feeding the input image into a dual-frequency feature extraction module, constructing the structural and textural dependencies of the single-view image through frequency-aware attention; integrating relevant information at different levels through a progressive stereo feature fusion module, further guiding the fusion of low-frequency and high-frequency information between the left and right views; finally, cascading the features from different levels of the left and right views and feeding them into the reconstruction part to obtain the final super-resolution image, and using frequency domain loss to balance high-frequency and low-frequency information. This invention effectively improves the quality of stereo super-resolution images while ensuring global consistency.

Description

Technical Field

[0001] This invention relates to the field of stereo image super-resolution technology, specifically a stereo image super-resolution method based on progressive feature extraction using frequency separation. Background Technology

[0002] The widespread use of stereoscopic display devices has made stereoscopic images increasingly common in our daily lives. With the increasing applications of stereoscopic images in 3D video, augmented reality, and virtual reality, the demand for high-resolution stereoscopic images is growing stronger. Furthermore, as a fundamental research area in computer vision, obtaining higher-quality stereoscopic image pairs will promote the development of depth estimation, stereo matching, 3D reconstruction, stereo object segmentation, and 3D detection. However, the resolution of stereoscopic images is usually limited by the acquisition equipment, and degraded stereoscopic image pairs will severely affect the human visual experience and subsequent applications. Therefore, designing an effective stereoscopic image super-resolution algorithm to recover high-resolution stereoscopic image pairs from low-resolution stereoscopic image pairs is a challenge with both practical and academic value.

[0003] Super-resolution is the process of reconstructing one or more high-resolution images from an input low-resolution image. Due to the uncertainty of the super-resolution solution, this type of problem is inherently highly ill-posed, meaning there is no unique solution. Against this challenging backdrop, stereo image super-resolution, as an emerging research field, still relies heavily on single-image and multi-image super-resolution techniques in its fundamental research methods.

[0004] Single-image super-resolution focuses primarily on the utilization and reconstruction of information within an image. It typically reconstructs images using example-based and deep learning-based methods, introducing convolutional neural networks and attention mechanisms to enhance the recovery of overall structure and key details. For binocular networks, the correlation and complementarity of information between views are more important. Under this premise, auxiliary information from multiple images can provide more references for binocular super-resolution design. Multi-image super-resolution reconstructs a single image from multiple images. Depending on the image acquisition method, it can be divided into video super-resolution, light field image super-resolution, and stereo image super-resolution. Multi-image super-resolution essentially utilizes block repetition and pixel offset between images to capture their correlation. However, video and optical flow super-resolution cannot capture sub-pixel correlations, which is detrimental to resolution improvement. Furthermore, multi-frame image processing expands the temporal dimension, primarily focusing on motion compensation, while binocular research focuses on disparity compensation. Different pixel offset methods also determine different correction methods. Existing stereo super-resolution methods mainly address disparity variations by extending the disparity attention module to achieve cross-view interaction. In addition, there are stereo super-resolution methods based on stereo matching disparity estimation, which jointly optimize disparity estimation and super-resolution to establish a correlation between left and right view information. However, current stereo super-resolution methods do not emphasize the importance of different spatial frequency features of images in the image restoration process, resulting in insufficient utilization of spatial and frequency information within the views. Furthermore, there is insufficient stereo interaction between views, with only single or simple multiple interactions between views, failing to effectively utilize relevant information from another view to restore the current view, and ignoring the potential for correlation between stereo image pairs.

[0005] Based on the above, this invention proposes a stereo image super-resolution method based on progressive feature extraction using frequency separation. Summary of the Invention

[0006] The purpose of this invention is to propose a stereo image super-resolution method based on progressive feature extraction with frequency separation to solve the problems of insufficient recovery of texture detail information and insufficient interaction between views in the existing technology when performing stereo image super-resolution.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A stereo image super-resolution method based on progressive feature extraction using frequency separation includes the following steps:

[0009] S1. The high-resolution image is downsampled by bicubic interpolation to obtain a low-resolution image. Then, the low-resolution image is divided into 30×90 blocks and fed into the network for training.

[0010] S2. The input image is fed into the dual-frequency feature extraction module, and the structural and texture dependencies of the single-view image are constructed through frequency-aware attention.

[0011] S3. Through the progressive stereo feature fusion module, relevant information at different levels is integrated to further guide the fusion of low-frequency and high-frequency information between the left and right views, thereby simulating the frequency perception of the human visual system from coarse to fine.

[0012] S4. Concatenate the features of different levels in the left and right views and then feed them into the reconstruction part to obtain the final super-resolution image.

[0013] Preferably, the workflow of the dual-frequency feature extraction module in S2 is as follows:

[0014] For the input image features, the high-frequency and low-frequency components of the input image are separated by using a fast Fourier transform and setting a filter with an appropriate threshold.

[0015] For relatively smooth low-frequency information, residual blocks are used to enhance the low-frequency structure and alleviate the smoothing problem; for high-frequency information, dense residual blocks are used to aggregate the high-frequency information, and then the high and low frequency information is converted back to the spatial domain through inverse fast Fourier transform.

[0016] The frequency-aware attention module is used to enhance the interaction between high-frequency and low-frequency information and full-frequency features extracted from residual dense groups, thereby strengthening the focus on low-frequency structural characteristics and high-frequency detailed features.

[0017] Preferably, the progressive stereo feature fusion module in S3 consists of low-frequency stereo cross-attention and high-frequency stereo cross-attention blocks, and its workflow is as follows:

[0018] First, multi-scale is introduced during the fusion of low-frequency information from the left and right views to enhance the structural feature representation between the two views;

[0019] Subsequently, enhanced low-frequency features are used to facilitate the fusion of high-frequency information between the two views, so as to preserve low-frequency structure while reconstructing high-frequency details.

[0020] Preferably, the workflow of the reconstruction section in S4 is as follows:

[0021] Preliminary hierarchical information fusion is performed using a convolutional layer with a kernel size of 1×1. Deep fusion of hierarchical information is then performed using a residual dense group consisting of 4 residual dense blocks, and finally fed into a sub-pixel layer to generate a super-resolution image.

[0022] Compared with existing technologies, this invention provides a stereo image super-resolution method based on progressive feature extraction through frequency separation, which has the following advantages:

[0023] This invention constructs the structural and detail dependencies of a single image based on frequency separation, making full use of the different frequency features of the image; and mimics the coarse-to-fine mechanism of the human visual system, guiding binocular high-frequency fusion through binocular low-frequency information fusion, and ensuring binocular vision through multiple interactions. Figure 1 By mining consistency and correlation information, high-quality stereo images can be effectively restored. In addition, the present invention designs a more targeted loss function, which balances the utilization of spatial and frequency domain information of the image and stabilizes the training process of the model. This ensures that it can be used in technical practice, such as in a wide range of image restoration tasks such as stereo image deblurring and low-light stereo image enhancement. The restoration effect of the method proposed in this invention meets the requirements of global semantic consistency and visual realism of the human eye, and has important value. Attached Figure Description

[0024] Figure 1 This is a flowchart of the stereo image super-resolution method based on progressive feature extraction using frequency separation proposed in this invention.

[0025] Figure 2 This is a structural diagram of the frequency-aware attention module mentioned in Embodiment 1 of the present invention;

[0026] Figure 3 This is a structural diagram of the low-frequency stereo cross-attention module mentioned in Embodiment 1 of the present invention;

[0027] Figure 4 This is a structural diagram of the low-frequency stereo cross-attention module mentioned in Embodiment 1 of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0029] Example 1:

[0030] Please see Figure 1 This invention proposes a stereo image super-resolution method based on progressive feature extraction using frequency separation, which includes the following steps:

[0031] 101: Preprocessing

[0032] High-resolution images are downsampled using bicubic interpolation to obtain low-resolution images of ×2 and ×4 resolution. These images are then divided into 30×90 blocks and fed into the network for training. Shallow features are extracted through 3×3 convolution.

[0033] 102: Dual-frequency feature extraction and restoration results

[0034] This embodiment constructs a dual-frequency feature extraction module. First, by setting a filter with an appropriate threshold, high-frequency and low-frequency components are separated. Second, different extraction modes are adopted for features of different frequencies. Finally, a frequency-aware attention module interacts with the spatial full-frequency information to emphasize the structural and detail dependencies of a single view. Specifically, the frequency-aware attention module employs an efficient attention mechanism to acquire the correlation information between high and low frequencies, which is then multiplied with the full-frequency information to obtain a feature map that emphasizes structure and detail.

[0035] 103: Progressive Stereo Interactive Restoration Results

[0036] This embodiment proposes a progressive stereoscopic interaction module that mimics the human visual system, enhancing the effectiveness of the entire process from top to bottom. First, multi-scale information is extracted through convolutional layers with multiple kernels of different sizes. Then, a cross-attention mechanism is used to fuse low-frequency structural information from both views. This information is then fed into a high-frequency fusion module to guide the fusion of high-frequency detail information from both views. Finally, joint attention is introduced to fuse high and low-frequency information, resulting in an image with higher consistency and fidelity to the real image.

[0037] 104: Multi-level interaction results

[0038] This embodiment proposes a multi-level stereo image feature interaction network, which cascades multiple stereo features to progressively improve the quality of super-resolution images from coarse to fine. Each level consists of two dual-frequency feature extraction modules and a progressive stereo interaction module, making full use of cross-view stereo features. Furthermore, this invention introduces FFT loss during the training process to balance the influence of high-frequency and low-frequency information.

[0039] 105: Technology Application

[0040] This embodiment, while ensuring global consistency in stereo image super-resolution, further improves the clarity of high-frequency textures. The progressive stereo image super-resolution method based on frequency separation, as an important research direction in the field of artificial intelligence, has demonstrated significant application value in multiple fields. In reality, this method has been widely applied to various products, such as virtual reality and 3D film production. By accessing this super-resolution function, users can easily improve the resolution of stereo images and obtain a more realistic visual experience. In addition, image processing manufacturers can also use this technology to optimize stereo image quality and enhance product competitiveness. Stereo image super-resolution methods have become a key technology in the current field of digital image processing, and its research results have profound social significance and application prospects.

[0041] Example 2:

[0042] Please see Figure 2-4Based on Example 1, but with some differences, the scheme in Example 1 will be further described below with specific calculation formulas and example data, as detailed in the following description:

[0043] 201: Preprocessing

[0044] (1) During training, bicubic interpolation was used to generate low-resolution images with 2x and 4x super-resolution. Then, these images were divided into blocks of size 30×90, and a cropping method with a stride of 20 was used to crop the high-resolution image blocks accordingly.

[0045] (2) In order to make full use of the dataset and increase the generalization ability of the model, the image patches used for training were randomly flipped in the horizontal and vertical directions.

[0046] 202: Dual-frequency feature extraction and restoration results

[0047] First, using Fast Fourier Transform (FFT), and by setting an appropriate threshold filter, the high-frequency and low-frequency components of the image patch are separated. Taking the left view as an example, we have:

[0048]

[0049] in, It refers to the low-frequency and high-frequency information obtained after frequency separation by Fast Fourier Transform.

[0050] Then, different methods are used to extract dual-frequency information based on its different characteristics. To alleviate the smoothing problem of low-frequency information, residual blocks are used to make the network focus more on strengthening the low-frequency structure, and then the information is returned to the spatial domain through inverse Fourier transform, i.e.:

[0051]

[0052] in, Res(·) represents the residual block.

[0053] Due to the sparsity of high-frequency details in the high-frequency subbands, dense residual blocks are used to aggregate high-frequency information while maintaining structural consistency, i.e.:

[0054]

[0055] in, RBD(·) represents a dense residual block.

[0056] Finally, the frequency domain modulation process begins. Specifically, first, dense residual groups are used to extract the full-frequency information of the image. This information is then fed as input to the high-frequency and low-frequency domain perceptual attention modules, respectively, to obtain feature maps emphasizing low-frequency structures and high-frequency details.

[0057]

[0058] In the frequency-domain aware attention module, firstly, a pixel-wise cross-channel context is aggregated by applying a 1×1 convolution, and then a 3×3 depthwise convolution is applied to encode the channel-wise spatial context, resulting in the generation of query (Q) key (K) and value (V) projections. Finally, we transpose Q and K, causing their dot products to interact and generate the attention map. In summary, the process is as follows:

[0059] Attention(Q,K,V)=V·Softmax(K·Q / α)(7)

[0060] 203: Progressive Stereo Interactive Restoration Results

[0061] This invention designs a progressive stereoscopic interaction module to fuse two views, which includes a low-frequency stereoscopic cross-attention module and a high-frequency stereoscopic cross-attention module.

[0062] The low-frequency stereo cross-attention module consists of a multi-scale module and a stereo cross-attention module. Given left and right features L and R, in the multi-scale module, a 1×1 convolution is first used to adjust the number of channels in the input feature map. Then, four convolutions with different kernel sizes (1×1, 3×3, 5×5, 7×7) are used to capture low-frequency information at different scales. Finally, the output features at different scales are concatenated to perform multi-scale information interaction. Taking the left view as an example, the above process is as follows:

[0063] L 1 =Conv 1 (Conv(L))(8)

[0064] L 3 =Conv 3 (Conv(L))(9)

[0065] L 5 =Conv 5 (Conv(L))(10)

[0066] L 7 =Conv 7 (Conv(L))(11)

[0067] L'=Conv(Cat(L 1 L 3 L 5 L 7 ))(12)

[0068] In the stereo cross-attention module, this invention employs scaled dot product attention, given left and right features. Obtain cross-view low-frequency fusion features as follows:

[0069]

[0070] Here, MS refers to the multi-scale feature extraction module, and LN refers to layer normalization.

[0071] Then the low-frequency information in the cross view Fusion with low-frequency information within the view:

[0072]

[0073] Where, γ L and γ R It is a trainable channel scale, initialized to zero for stable training.

[0074] The high-frequency stereo cross-attention module consists of a low-frequency guiding high-frequency module and a stereo cross-attention module. In the guiding module (Low-to-High Frequency Guiding, LHFG), two modulation parameters, γ and β, are used to adaptively control this process. First, low-frequency information is fused... The input is fed into two 1×1 convolutions to obtain γ and β, which are then transformed by scaling and shifting the feature maps of specific layers. Take the left view as an example:

[0075]

[0076] Where ⊙ represents element-wise product, and I represents sum. Tensors with the same dimension and a value of 1. The right-view process is similar.

[0077] The stereo cross-attention module is similar to the low-frequency module:

[0078]

[0079] Finally, the low-frequency output information and the high-frequency output information are combined. The channels are concatenated along the channel dimension, then passed through a joint attention module, and finally transformed back to their original dimensions using a 1×1 convolution, completing the entire fusion operation.

[0080] 204: Multi-level Interaction Results

[0081] Since hierarchical features have proven effective for stereo tasks, this invention alternates between multiple dual-frequency feature extraction modules and progressive feature interaction modules to extract hierarchical features from stereo images. First, the hierarchical features from the left and right views are concatenated and fed into the reconstruction section, where a 1×1 convolutional layer is used for initial hierarchical information fusion. Then, a dense residual group consisting of four dense residual blocks is used for deep fusion of the hierarchical information. Finally, the data is fed into a sub-pixel layer to generate a super-resolution image.

[0082] Example 3:

[0083] Based on Examples 1 and 2, but with some differences, the feasibility of the schemes in Examples 1 and 2 is verified below with specific experiments, as detailed in the following description:

[0084] This experiment evaluates the performance of all methods on four benchmark stereo image datasets, including 20 pairs of stereo images from the KITTI2012 dataset, 20 pairs of stereo images from the KITTI2015 dataset, 5 pairs of stereo images from the Middlebury dataset, and 112 pairs of stereo images from the Flickr1024 dataset.

[0085] To comprehensively evaluate the quality of stereo super-resolution images, we employed two objective metrics: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). PSNR measures the similarity between the reconstructed image and the original image. SSIM calculates the structural similarity between different images. Higher PSNR and SSIM values ​​indicate better image restoration performance.

[0086] To verify the performance of this invention, the model constructed in this invention was compared with 11 state-of-the-art image SR methods: 1) Single-image SR methods: VDSR, EDSR, RDN, and RCAN; 2) Stereo-image SR methods: StereoSR, PASSRnet, SRRes+SAM, IMSSRnet, BSSRnet, iPASSRnet, and SSRDE-FNet. All compared methods were trained on the same training dataset as the method proposed in this invention to ensure a fair comparison. The results are shown in Tables 1 and 2, and the calculated metrics are all left view + right view / 2.

[0087] Quantitative analysis under Table 1×4

[0088]

[0089]

[0090] Quantitative analysis under Table 2×2

[0091]

[0092]

[0093] Tables 1 and 2 list the super-resolution results of this invention at ×4 and ×2 scales in four databases. In each column, the best-performing result is shown in bold. As can be seen from the tables, this invention outperforms the compared single-image and stereo image super-resolution methods in PSNR and SSIM at both scales, and has a smaller number of model parameters. Overall, this invention can recover high-quality stereo images while maintaining high performance.

[0094] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A stereo image super-resolution method based on progressive feature extraction using frequency separation, characterized in that, Includes the following steps: S1. The high-resolution image is downsampled by bicubic interpolation to obtain a low-resolution image. Then, the low-resolution image is divided into 30×90 blocks and fed into the network for training. S2. The input image is fed into the dual-frequency feature extraction module, which constructs the structural and texture dependencies of the single-view image through frequency-aware attention. The specific workflow of the dual-frequency feature extraction module is as follows: For the input image features, the high-frequency and low-frequency components of the input image are separated by using a fast Fourier transform and setting a filter with an appropriate threshold. For relatively smooth low-frequency information, residual blocks are used to enhance the low-frequency structure and alleviate the smoothing problem; for high-frequency information, dense residual blocks are used to aggregate the high-frequency information, and then the high and low frequency information is converted back to the spatial domain through inverse fast Fourier transform. The frequency-aware attention module is used to enhance the interaction between high-frequency and low-frequency information and full-frequency features extracted from residual dense groups, thereby strengthening the focus on low-frequency structural characteristics and high-frequency detailed features. S3. Through a progressive stereo feature fusion module, relevant information at different levels is integrated to further guide the fusion of low-frequency and high-frequency information between the left and right views, thereby simulating the human visual system's frequency perception from coarse to fine. The progressive stereo feature fusion module consists of low-frequency stereo cross-attention and high-frequency stereo cross-attention blocks, and its workflow is as follows: First, multi-scale is introduced during the fusion of low-frequency information from the left and right views to enhance the structural feature representation between the two views; Subsequently, enhanced low-frequency features are used to facilitate the fusion of high-frequency information between the two views, so as to preserve low-frequency structure while reconstructing high-frequency details; S4. The features of different levels in the left and right views are concatenated and then fed into the reconstruction part to obtain the final super-resolution image; the specific workflow of the reconstruction part is as follows: Preliminary hierarchical information fusion is performed using a convolutional layer with a kernel size of 1×1. Deep fusion of hierarchical information is then performed using a residual dense group consisting of 4 residual dense blocks, and finally fed into a sub-pixel layer to generate a super-resolution image.

Images