Stereo image super-resolution method for progressive feature extraction based on frequency separation

Abstract

The invention discloses a stereo image super-resolution method for progressive feature extraction based on frequency separation, and particularly relates to the technical field of image super-resolution. The method provided by the invention comprises the following steps: preprocessing a low-resolution image; the input image is sent to a dual-frequency feature extraction module, and the structural and texture dependency relationship of a single-view image is constructed through frequency perception attention; integrating related information of different levels through a progressive three-dimensional feature fusion module, and further guiding the fusion of low-frequency and high-frequency information between the left and right views; and finally, features of different levels of the left view and the right view are cascaded and then sent to a reconstruction part, a final super-resolution image is obtained, and high-frequency and low-frequency information is balanced by using frequency domain loss. According to the method, the quality of the three-dimensional super-resolution image is effectively improved on the basis of ensuring the global consistency.

Description

Technical Field

[0001] The present invention relates to the technical field of stereoscopic image super-resolution, and in particular to a stereoscopic image super-resolution method based on progressive feature extraction of frequency separation. Background Art

[0002] The widespread use of stereoscopic display devices has made stereoscopic images more and more common in our daily lives. With the increasing application of stereoscopic images in 3D video, augmented reality, virtual reality and other practical aspects, the demand for high-resolution stereoscopic images has become increasingly strong. In addition, as a basic research in the field of computer vision, obtaining higher-quality stereo image pairs will also promote the development of depth estimation, stereo matching, three-dimensional reconstruction, stereo object segmentation, 3D detection and other aspects. However, the resolution of stereo images is usually limited by the acquisition equipment, and the degraded stereo image pairs will seriously affect people's visual experience and subsequent applications. Therefore, designing an effective stereo image super-resolution algorithm to recover high-resolution stereo image pairs from low-resolution stereo image pairs is a challenge with both practical and academic value.

[0003] The super-resolution problem is the process of reconstructing an input low-resolution image into one or more high-resolution images. Due to the uncertainty of the super-resolution image solution, this type of problem is inherently highly ill-posed, that is, there is no unique solution. In such a challenging context, stereo image super-resolution, as an emerging research field, its basic research methods still rely heavily on single-image super-resolution and multi-image super-resolution techniques.

[0004] Single-image super-resolution focuses on the utilization and reconstruction of information within the image. It generally reconstructs images through sample-based and deep learning-based methods, and introduces convolutional neural networks and attention mechanisms to enhance the recovery of overall structure and key details. For binocular networks, information correlation and complementarity between views are more important. Under this premise, the auxiliary information of multiple images can provide more references for binocular super-resolution design. Multi-image super-resolution reconstructs a single image through multiple images. According to the different image acquisition methods, it can be divided into video super-resolution, light field image super-resolution, and stereo image super-resolution. Multi-image super-resolution essentially uses block repetition and pixel offset between images to capture their correlation, but video and optical flow super-resolution cannot obtain the correlation between sub-pixels, which is not conducive to the improvement of resolution. At the same time, the processing of multi-frame images expands the time dimension and mainly studies motion compensation, while the research focus of binocular is disparity compensation. Different pixel offset methods also determine different correction methods. Existing stereo super-resolution methods mainly respond to disparity changes by expanding the disparity attention module to achieve cross-view interaction. There are also stereo super-resolution methods based on stereo matching disparity estimation, which jointly optimize disparity estimation and super-resolution to establish information association between left and right views. However, current stereo super-resolution methods do not pay attention to the importance of different spatial frequency features of images in the image restoration process, resulting in insufficient utilization of spatial and frequency information within the view. In addition, there is insufficient stereo interaction between views, with only a single or simple multiple interactions between views, and the relevant information of another view cannot be effectively used to restore the current view, ignoring the potential of correlation between stereo image pairs.

[0005] Based on the above content, the present invention proposes a stereo image super-resolution method based on progressive feature extraction of frequency separation. Summary of the invention

[0006] The purpose of the present invention is to propose a stereo image super-resolution method based on frequency separation and progressive feature extraction to solve the problems of insufficient restoration of texture detail information and insufficient interaction between views in the prior art during stereo image super-resolution.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A stereo image super-resolution method based on progressive feature extraction of frequency separation comprises the following steps:

[0009] S1, downsample the high-resolution image by bicubic interpolation to obtain a low-resolution image, then divide the low-resolution image into 30×90 blocks and send it to the network for training;

[0010] S2, feed the input image into the dual-frequency feature extraction module, and construct the structural and texture dependencies of the single-view image through frequency-aware attention;

[0011] S3, through the progressive stereo feature fusion module, integrates relevant information at different levels, further guides the fusion of low-frequency and high-frequency information between the left and right views, and then simulates the frequency perception of the human visual system from coarse to fine;

[0012] S4: Concatenate the features of different levels of the left and right views and send them to the reconstruction part to obtain the final super-resolution image.

[0013] Preferably, the workflow of the dual-frequency feature extraction module in S2 is specifically as follows:

[0014] For the input image features, the high-frequency and low-frequency components of the input image are separated by fast Fourier transform and setting filters with appropriate thresholds;

[0015] For relatively smooth low-frequency information, residual blocks are used to enhance the low-frequency structure and alleviate the smoothing problem; for high-frequency information, dense residual blocks are used to aggregate high-frequency information, and then, the high and low-frequency information is converted back to the spatial domain through inverse fast Fourier transform;

[0016] A frequency-aware attention module is used to promote the interaction enhancement between high-frequency and low-frequency information and the full-frequency features extracted by the residual dense group to strengthen the attention to low-frequency structural characteristics and high-frequency detail features.

[0017] Preferably, the progressive stereo feature fusion module in S3 is composed of low-frequency stereo cross attention blocks and high-frequency stereo cross attention blocks, and its workflow is specifically as follows:

[0018] Firstly, multi-scale is introduced in the process of fusing low-frequency information from the left view and the right view to enhance the structural feature representation between the two views.

[0019] Subsequently, the enhanced low-frequency features are used to promote the fusion of high-frequency information between the two views, ensuring that the low-frequency structure is preserved while reconstructing the high-frequency details.

[0020] Preferably, the workflow of the reconstruction part in S4 is specifically as follows:

[0021] A convolutional layer with a kernel size of 1×1 is used for preliminary fusion of hierarchical information, and a residual dense group consisting of 4 residual dense blocks is used for deep fusion of hierarchical information, which is then fed into a sub-pixel layer to generate a super-resolution image.

[0022] Compared with the prior art, the present invention provides a stereo image super-resolution method based on progressive feature extraction of frequency separation, which has the following beneficial effects:

[0023] The present invention constructs the structural dependence and detail dependence of a single image based on frequency separation, making full use of the different frequency characteristics of the image; and imitates the coarse-to-fine mechanism of the human visual system, guides binocular high-frequency fusion through binocular low-frequency information fusion, and ensures binocular vision through multiple interactions. Figure 1 The consistency and correlation information is mined to effectively restore high-quality stereo images; in addition, the present invention designs a more targeted loss function, balances the utilization of image spatial and frequency domain information, and stabilizes the training process of the model, which ensures that it can be used in technical practice. For example, in a wide range of image restoration tasks such as stereo image deblurring and low-light stereo image enhancement, the restoration effect of the method proposed in the present invention meets the global semantic consistency and visual authenticity requirements of the human eye, and is of great value. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a stereo image super-resolution method based on progressive feature extraction of frequency separation proposed by the present invention;

[0025] Figure 2 This is a structural diagram of the frequency perception attention module mentioned in Example 1 of the present invention;

[0026] Figure 3 This is a structural diagram of the low-frequency stereo cross-attention module mentioned in Example 1 of the present invention;

[0027] Figure 4 This is a structural diagram of the low-frequency stereo cross-attention module mentioned in Example 1 of the present invention. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0029] Embodiment 1:

[0030] See also Figure 1 The present invention proposes a stereo image super-resolution method based on progressive feature extraction of frequency separation, the method comprising the following steps:

[0031] 101: Preprocessing

[0032] The high-resolution images are downsampled by bicubic interpolation to obtain low-resolution images of ×2 and ×4, and these images are divided into 30×90 blocks and sent to the network for training. Shallow features are extracted through 3×3 convolution.

[0033] 102: Dual-frequency feature extraction and restoration results

[0034] This embodiment constructs a dual-frequency feature extraction module. First, by setting a filter with an appropriate threshold, the high-frequency component and the low-frequency component are separated. Secondly, different extraction modes are adopted for the features of different frequency information. Finally, the frequency-aware attention module interacts with the spatial full-frequency information to emphasize the structural dependence and detail dependence of the single view. Among them, an efficient attention mechanism is used in the frequency-aware attention module to obtain the correlation information of high and low frequencies, and then interactively multiply it with the full-frequency information to obtain a feature map that emphasizes structure and details.

[0035] 103: Progressive Stereo Interaction Restoration Results

[0036] This embodiment proposes a progressive stereo interaction module that imitates the human visual system to enhance the effectiveness of the entire process from top to bottom. First, multi-scale information is extracted through convolutional layers with multiple different convolution kernels, and the low-frequency structural information of the two views is fused using the cross-attention mechanism. Then, it is sent to the high-frequency fusion information module to guide the fusion of the high-frequency detail information of the two views. Finally, joint attention is introduced to fuse the high- and low-frequency information, so that the generated image has higher consistency and fidelity with the real image.

[0037] 104: Multi-level interaction results

[0038] This embodiment proposes a multi-level stereo image feature interaction network composed of multiple stereo features interacting and cascading, gradually improving the quality of super-resolution images from coarse to fine. Each level consists of two dual-frequency feature extraction modules and a progressive stereo interaction module, making full use of cross-view stereo features. In addition, the present invention introduces FFT loss in the training process to balance the influence of high-frequency and low-frequency information.

[0039] 105: Technology Application

[0040] This embodiment better improves the clarity of high-frequency textures on the basis of ensuring the global consistency of stereoscopic image super-resolution. As an important research direction in the field of artificial intelligence, the progressive stereoscopic image super-resolution method based on frequency separation has demonstrated significant application value in many fields. In reality, this method has been widely used in various products, such as virtual reality, 3D movie production, etc. By accessing the super-resolution function, users can easily improve the resolution of stereoscopic images and obtain a more realistic visual experience. In addition, image processing manufacturers can also use this technology to optimize the quality of stereoscopic images and enhance product competitiveness. The stereoscopic image super-resolution method has become a key technology in the current field of digital image processing, and its research results have far-reaching social significance and application prospects.

[0041] Embodiment 2:

[0042] See also Figure 2-4, based on Example 1 but different in that the solution in Example 1 is further introduced below in combination with specific calculation formulas and example data, as described below:

[0043] 201: Preprocessing

[0044] (1) During the training process, bicubic interpolation is used to generate low-resolution images with 2x and 4x super-resolution. These images are then segmented into blocks of size 30×90 and cropped with a step size of 20, while the high-resolution image blocks are cropped accordingly.

[0045] (2) In order to make full use of the dataset and increase the generalization ability of the model, the training image patches are randomly flipped horizontally and vertically.

[0046] 202: Dual-frequency feature extraction recovery results

[0047] First, use the fast Fourier transform to separate the high-frequency and low-frequency components of the image block by setting a filter with an appropriate threshold. Taking the left view as an example, we have:

[0048]

[0049] in, Refers to the low-frequency and high-frequency information obtained after fast Fourier transform frequency separation.

[0050] Then, different methods are used to extract dual-frequency information according to the different characteristics of high- and low-frequency information. In order to alleviate the problem of low-frequency information smoothing, a residual block is used to make the network focus more on strengthening the low-frequency structure, and then return it to the spatial domain through the inverse Fourier transform, that is:

[0051]

[0052] in, Res(·) represents the residual block.

[0053] Due to the sparsity of high-frequency details in the high-frequency subband, a dense residual block is used to aggregate high-frequency information to maintain structural consistency, namely:

[0054]

[0055] in, RBD(·) denotes dense residual block.

[0056] Finally, we enter the frequency domain modulation process. Specifically, we first use a dense residual group to extract the full-frequency information of the image, and then feed it into the high-frequency and low-frequency frequency domain perception attention modules as inputs to obtain feature maps that emphasize low-frequency structures and high-frequency details respectively:

[0057]

[0058] In the frequency-aware attention module, we first aggregate pixel-by-pixel cross-channel context by applying 1×1 convolutions, and then apply 3×3 depthwise convolutions to encode channel-by-channel spatial contexts to generate query (Q) key (K) and value (V) projections. Finally, we transpose Q and K so that their dot product interacts to generate the attention map. In general, the process is as follows:

[0059] Attention(Q,K,V)=V·Softmax(K·Q / α)(7)

[0060] 203: Progressive Stereo Interaction Restoration Results

[0061] The present invention designs a progressive stereo interaction module to fuse dual views, which includes a low-frequency stereo cross-attention module and a high-frequency stereo cross-attention module.

[0062] The low-frequency stereo cross-attention module consists of a multi-scale module and a stereo cross-attention module. Given the left and right features L and R, in the multi-scale module, a 1×1 convolution is first used to adjust the number of channels of the input feature map. Then, four convolutions with different kernel sizes (1×1, 3×3, 5×5, 7×7) are used to capture low-frequency information of different scales. Finally, the output features of different scales are connected for multi-scale information interaction. Taking the left view as an example, the above process is as follows:

[0063] L 1 =Conv 1 (Conv(L))(8)

[0064] L 3 =Conv 3 (Conv(L))(9)

[0065] L 5 =Conv 5 (Conv(L))(10)

[0066] L 7 =Conv 7 (Conv(L))(11)

[0067] L'=Conv(Cat(L 1 , L 3 , L 5 , L 7 ))(12)

[0068] In the stereo cross-attention module, the present invention adopts scaled dot product attention, given the left and right features Get cross-view low-frequency fusion features as follows:

[0069]

[0070] Among them, MS refers to the multi-scale feature extraction module and LN refers to layer normalization.

[0071] Then the low frequency information in the cross view Fusion with low-frequency information within the view:

[0072]

[0073] Among them, γ L and γ R is the trainable channel scale and is initialized to zero to stabilize training.

[0074] The high-frequency stereo cross-attention module consists of a low-frequency guiding high-frequency module and a stereo cross-attention module. In the guidance module (Low-to-High Frequency Guiding, LHFG), two modulation parameters γ and β are used to adaptively control this process. First, the low-frequency fusion information It is fed into two 1×1 convolutions to obtain γ and β, and then transformed by scaling and shifting the feature maps of specific layers. Take the left view as an example:

[0075]

[0076] Among them, ⊙ represents the element product, I is the sum A tensor with the same dimension and value of 1. The process is similar for the right view.

[0077] The stereo cross-attention module is similar to the low-frequency one:

[0078]

[0079] Finally, the low-frequency output information is combined with the high-frequency output information The channels are concatenated in the channel dimension and then passed through the joint attention module and 1×1 convolution to transform the channels into their original dimensions, completing the entire fusion operation.

[0080] 204: Multi-level interaction results

[0081] Since hierarchical features have been proven to be effective for stereo tasks, the present invention alternately uses multiple dual-frequency feature extraction modules and progressive feature interaction modules to extract hierarchical features of stereo images. First, the hierarchical features of the left and right views are concatenated and fed into the reconstruction part, and a convolutional layer with a kernel size of 1×1 is used for preliminary hierarchical information fusion. Then, a dense residual group consisting of 4 dense residual blocks is used for deep fusion of hierarchical information. Finally, it is fed into a sub-pixel layer to generate a super-resolution image.

[0082] Embodiment 3:

[0083] Based on Examples 1-2, but with the difference that the feasibility of the solutions in Examples 1 and 2 is verified in combination with specific experiments, as described below:

[0084] This experiment evaluates the performance of all methods on four benchmark stereo image datasets, including 20 pairs of stereo images from the KITTI2012 dataset, 20 pairs of stereo images from the KITTI2015 dataset, 5 pairs of stereo images from the Middlebury dataset, and 112 pairs of stereo images from the Flickr1024 dataset.

[0085] In order to comprehensively evaluate the quality of stereo super-resolution images, we use two objective indicators: Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM). PSNR is used to measure the similarity between the reconstructed image and the original image. SSIM calculates the structural similarity between different images. The higher the PSNR and SSIM values, the better the image restoration performance.

[0086] In order to verify the performance of the present invention, the model constructed by the present invention is compared with 11 state-of-the-art image SR methods: 1) Single image SR methods: VDSR, EDSR, RDN and RCAN; 2) Stereo image SR method 1: StereoSR, PASSRnet, SRRes+SAM, IMSSRnet, BSSRnet, iPASSRnet, SSRDE-FNet. All compared methods are trained on the same training dataset as the method proposed in the present invention for fair comparison. The results are shown in Tables 1 and 2, and the calculated indicators are all left view + right view / 2.

[0087] Quantitative analysis under Table 1×4

[0088]

[0089]

[0090] Quantitative analysis under table 2×2

[0091]

[0092]

[0093] Tables 1 and 2 list the super-resolution results of the present invention in four databases at ×4 and ×2. In each column, the best performance result is shown in bold. It can be seen from the table that the PSNR and SSIM of the present invention at two magnifications are better than the compared single image super-resolution and stereo image super-resolution methods, and the model parameters are small. In general, the present invention can restore high-quality stereo images while ensuring efficient performance.

[0094] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A stereo image super-resolution method based on progressive feature extraction of frequency separation, characterized in that: The following steps are involved: S1, downsample the high-resolution image by bicubic interpolation to obtain a low-resolution image, then divide the low-resolution image into 30×90 blocks and send it to the network for training; S2, feed the input image into the dual-frequency feature extraction module, and construct the structural and texture dependencies of the single-view image through frequency-aware attention; S3, through the progressive stereo feature fusion module, integrates relevant information at different levels, further guides the fusion of low-frequency and high-frequency information between the left and right views, and then simulates the frequency perception of the human visual system from coarse to fine; S4: Concatenate the features of different levels of the left and right views and send them to the reconstruction part to obtain the final super-resolution image.

2. The method for stereoscopic image super-resolution based on progressive feature extraction of frequency separation according to claim 1, characterized in that: The specific working process of the dual-frequency feature extraction module in S2 is as follows: For the input image features, the high-frequency and low-frequency components of the input image are separated by fast Fourier transform and setting filters with appropriate thresholds; For relatively smooth low-frequency information, residual blocks are used to enhance the low-frequency structure and alleviate the smoothing problem; for high-frequency information, dense residual blocks are used to aggregate high-frequency information, and then, the high and low-frequency information is converted back to the spatial domain through inverse fast Fourier transform; A frequency-aware attention module is used to promote the interaction enhancement between high-frequency and low-frequency information and the full-frequency features extracted by the residual dense group to strengthen the attention to low-frequency structural characteristics and high-frequency detail features.

3. The stereoscopic image super-resolution method based on progressive feature extraction of frequency separation according to claim 1, characterized in that: The progressive stereo feature fusion module described in S3 consists of low-frequency stereo cross-attention and high-frequency stereo cross-attention blocks, and its workflow is as follows: Firstly, multi-scale is introduced in the process of fusing low-frequency information from the left view and the right view to enhance the structural feature representation between the two views. Subsequently, the enhanced low-frequency features are used to promote the fusion of high-frequency information between the two views, ensuring that the low-frequency structure is preserved while reconstructing the high-frequency details.

4. The stereo image super-resolution method based on progressive feature extraction of frequency separation according to claim 1, characterized in that: The specific workflow of the reconstruction part described in S4 is as follows: A convolutional layer with a kernel size of 1×1 is used for preliminary fusion of hierarchical information, and a residual dense group consisting of 4 residual dense blocks is used for deep fusion of hierarchical information, which is then fed into a sub-pixel layer to generate a super-resolution image.

Images