Single image super-resolution reconstruction method and system based on wavelet transform and cross-domain feature fusion

Abstract

The invention discloses a single image super-resolution reconstruction method and system based on wavelet transform and cross-domain feature fusion. According to the method, firstly, a low-resolution RGB image is mapped to a high-dimensional feature space through a shallow feature extraction module; performing up-sampling and discrete wavelet decomposition on the features by using a wavelet feature mixing module to obtain multi-band features; low-frequency and high-frequency depth features are respectively extracted through a double-branch structure, cross-domain fusion is realized by means of a deformable cross attention mechanism, and the feature expression ability is enhanced in combination with residual connection; and finally, reconstructing a high-resolution image through convolution, up-sampling and regularization processing. In the training process, a pixel-level loss function is adopted to optimize network parameters, the multi-frequency-domain feature sensitivity is effectively improved, texture and structure information is balanced, the image contrast, definition and structural integrity are improved, and high-quality real-time super-resolution reconstruction can be achieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a single image super-resolution reconstruction method and system based on wavelet transform and cross-domain feature fusion. BACKGROUND

[0002] Single Image Super-Resolution (SISR) is an important computer vision technology that aims to reconstruct high-quality high-resolution images from a single low-resolution image. This technology is widely used in medical imaging, remote sensing observation, video surveillance and other fields, and can improve the clarity and detail performance of images without the need to upgrade hardware devices. However, due to the information loss and degradation in low-resolution images, how to accurately recover high-frequency details and texture information has always been a core challenge in the super-resolution task.

[0003] Traditional super-resolution reconstruction methods are mainly based on interpolation algorithms (such as bicubic interpolation), reconstruction-based methods (such as iterative back projection) and learning-based methods (such as sparse coding and dictionary learning). These methods can improve image resolution to some extent, but when dealing with complex scenes and large-scale factors, it is often difficult to recover fine structures and edge information. In recent years, the progress of deep learning technology has significantly improved the performance of SISR. Methods based on Convolutional Neural Network (CNN), such as SRCNN and EDSR, perform well in extracting local high-frequency details, but are limited by small receptive fields and are difficult to capture global information. Models based on Transformer, such as SwinIR, make up for the shortcomings of CNN by modeling long-range dependencies and achieve the best results, but perform worse than CNN in high-frequency detail reconstruction, and the computational complexity grows quadratically with space.

[0004] Recently, Wavelet transform (WT) has also been proven to be an effective frequency domain tool that can capture image context and texture information at multiple scales, which is crucial for image detail recovery and has been applied to SISR. As DWSR simplifies the mapping from LR to HR by reconstructing wavelet coefficients, WGSR utilizes wavelet domain loss functions to train a Generative Adversarial Network (GAN)-based super-resolution image model. Despite some progress, existing wavelet-based methods still struggle to achieve comparable performance to Transformer-based super-resolution models. Their main limitations include limited receptive field, suboptimal utilization of high-frequency information, and insufficient exploration of the intrinsic relationship between different frequency domains. Addressing these challenges remains an open research direction for further improving wavelet-driven super-resolution frameworks.

[0005] Inspired by the above observations, the present application proposes a single image super-resolution reconstruction method framework based on wavelet transform and cross-domain feature fusion. SUMMARY

[0006] To solve the above technical problems, the present application provides a single image super-resolution reconstruction method and system based on wavelet transform and cross-domain feature fusion.

[0007] The technical scheme adopted by the present application to solve its technical problems is:

[0008] A single image super-resolution reconstruction method based on wavelet transform and cross-domain feature fusion, the method comprising the following steps:

[0009] S100: Obtain a low-resolution RGB image and construct a single image super-resolution network, which includes a shallow feature extraction module, a deep feature extraction module, and a high-quality image reconstruction stage in sequence, wherein the deep feature extraction module includes multiple wavelet feature mixing modules;

[0010] S200: Input the low-resolution RGB image into the shallow feature extraction module and use a convolution structure to convert the input image features, mapping it from the RGB space to a high-dimensional feature space to obtain a low-resolution shallow feature map;

[0011] S300: input the shallow feature map into the wavelet feature mixing module in the deep feature extraction module, first up-sample and discrete wavelet transform the shallow feature map to obtain a multi-band feature map; then perform deep feature extraction on different frequency band feature maps through a double-branch feature extraction structure to obtain low-frequency deep feature maps and high-frequency deep feature maps; then perform cross-domain fusion on the low-frequency deep feature maps and the high-frequency deep feature maps through a deformable cross-attention mechanism to obtain high-low frequency fusion features; repeat the above frequency band decomposition, double-branch extraction and cross-domain fusion operations along the network depth direction, and combine residual connections to obtain a deep feature map;

[0012] S400: perform convolution, up-sampling and overfitting suppression processing on the deep feature map to generate a high-resolution reconstructed image of a target scale;

[0013] S500: calculate the difference between the high-resolution reconstructed image and the real high-resolution image using a pixel-level loss function, optimize the network parameters based on the difference, and obtain an optimized single image super-resolution network to complete real-time single image super-resolution reconstruction.

[0014] Preferably, the convolution structure in S200 is a hierarchical convolution structure composed of a 7x7 convolution layer and a 3x3 convolution layer; wherein the 7x7 convolution layer is used to capture global context information of the image, and the 3x3 convolution layer is used to locally refine the captured features, and finally generate a low-resolution shallow feature map, specifically:

[0015] F S =H SF (I LR )

[0016] wherein I LR ∈R 3×h×w is a low-resolution image, h and w are the height and width of the image respectively, F S is a low-resolution shallow feature map, and H SF () represents a shallow feature extraction operation.

[0017] Preferably, the up-sampling multiple in S300 is two; the discrete wavelet transform decomposes the up-sampled feature map into a low-low frequency feature map LL, a low-high frequency feature map LH, a high-low frequency feature map HL and a high-high frequency feature map HH, and the sizes of the decomposed feature maps remain consistent with the input shallow feature map.

[0018] Preferably, the double-branch feature extraction structure in S300 includes a smooth feature extraction block SFEB based on a Transformer architecture and an edge feature extraction block EFEB based on a CNN architecture; SFEB is composed of a convolutional layer, a window multi-head self-attention module, a Swin Transformer module and an overlapping cross-attention module OCA connected in sequence, SFEB is used to process the low-low frequency feature map LL to obtain a low-frequency deep feature map, and its output expression is:

[0019]

[0020] wherein, is the input feature, H conv (·) represents a convolution operation, H win (·) represents a window multi-head self-attention operation, H swi n(·) represents a Swin Transformer processing operation, H OCA (·) represents an overlapping cross-attention operation, is the low-frequency deep feature map;

[0021] The window multi-head self-attention module divides the input feature map into non-overlapping local windows of MxM size, and independently performs self-attention calculation on each window, and the total number of windows is M is the window side length; the overlapping cross-attention module OCA adopts a sliding window strategy, and expands the window into an overlapping window of MoxMo size, and Mo is the overlapping window side length;

[0022] EFEB includes two convolutional layers with normalization and a channel attention mechanism based on average pooling, and EFEB is used to process the spliced features of the low-high frequency feature map LH, the high-low frequency feature map HL and the high-high frequency feature map HH to obtain a high-frequency deep feature map, and its output expression is:

[0023]

[0024] wherein, represents the input feature of EFEB, H Li (·) represents a linear transformation operation, H and W are the height and width of the feature map respectively, sigmoid(·) represents an activation function, is the high-frequency deep feature map.

[0025] Preferably, the low-frequency deep feature map and the high-frequency deep feature map are cross-domain fused by using a deformable cross-attention mechanism in S300 to obtain a high-low frequency fusion feature, which includes:

[0026] The low-frequency offset matrix and the high-frequency offset matrix are calculated by a low-frequency offset network and a high-frequency offset network respectively on the low-frequency depth feature map and the high-frequency depth feature map, and the deformable attention mask is calculated by a mask network on the spliced features of the low-frequency depth feature map and the high-frequency depth feature map.

[0027] Pixel-level optical flow distortion operations are performed on the low-frequency depth feature map and the high-frequency depth feature map based on the low-frequency offset matrix and the high-frequency offset matrix respectively, the query matrix Q and the value matrix V are constructed based on the low-frequency depth feature map after optical flow distortion and the deformable attention mask, the key matrix K is constructed based on the high-frequency depth feature map after optical flow distortion and the deformable attention mask, and the part not subjected to attention calculation is processed by 3*3 convolution.

[0028] Preferably, the calculation of the low-frequency offset matrix, the high-frequency offset matrix and the deformable attention mask is specifically as follows:

[0029]

[0030]

[0031] θ mask =H mask (concat(F L ,F H ))

[0032] Wherein, F L and F H are the low-frequency depth feature map and the high-frequency depth feature map respectively, and represent the low-frequency and high-frequency offset networks respectively, H mask (·) represents the mask network, concat(·,·) represents splicing along the channel dimension, and θ mask is the deformable attention mask, and are the low-frequency offset matrix and the high-frequency offset matrix respectively;

[0033] The deformable attention mask output by the mask network is used to screen out K windows for sparse attention calculation, and the remaining n-K windows are processed by convolution, wherein n is the total number of windows of the feature map, and the calculation formula is as follows: Ws is the window size; the attention calculation formula is as follows:

[0034]

[0035]

[0036] Q=θ mask ×F L ’×W q ,K=θmask ×F H ’×W k

[0037] V=θ mask ×F L ’×W v ,V S =(1-θ mask )×F L ’×W v

[0038]

[0039] where Flowwarp(·) denotes the optical flow warping operation, W q , W k and W v are the query matrix, key matrix and value matrix respectively, d represents the dimension of the query / key, B denotes the learnable relative position encoding, F L ’ is the low-frequency depth feature map after optical flow warping, F H ’ is the high-frequency depth feature map after optical flow warping, and is the output of the deformable cross-attention.

[0040] Preferably, the above frequency band decomposition, double-branch extraction and cross-domain fusion operations are repeated along the network depth direction in S300, and the residual connection is combined to obtain a deep feature map, specifically:

[0041]

[0042] wherein, denotes the target feature map output by the last deformable cross-attention module, denotes the target feature map output by the i-th deformable cross-attention module, denotes the operation of the i-th wavelet feature mixing module WFM, F R denotes the deep feature map, denotes the input of the first WFM module.

[0043] Preferably, the high-quality image reconstruction module in S400 is composed of a convolutional layer, an upsampling operation and a dropout layer, and S400 is specifically:

[0044] I SR =H HQIR (F R )

[0045] wherein H HQIR (·) denotes the high-quality image reconstruction operation, and I SR denotes the high-resolution reconstructed image.

[0046] Preferably, the pixel-level loss function in S500 is specifically:

[0047]

[0048] wherein i represents an image index, N represents the number of training pairs, I SE is an output obtained by taking I LR as the input of the network, and I HR is the corresponding true value, i.e., a high-quality image.

[0049] A single image super-resolution reconstruction system based on wavelet transform and cross-domain feature fusion, comprising:

[0050] A single image super-resolution network module is configured to obtain a low-resolution RGB image and construct a single image super-resolution network, which comprises a shallow feature extraction module, a deep feature extraction module and a high-quality image reconstruction stage in sequence; wherein the deep feature extraction module comprises a plurality of wavelet feature mixing modules.

[0051] The shallow feature extraction module is configured to convert the input low-resolution RGB image from RGB space to a high-dimensional feature space by using a convolution structure, thereby obtaining a low-resolution shallow feature map.

[0052] The deep feature extraction module is configured to upsample and discrete wavelet transform the shallow feature map to obtain a multi-band feature map; then extract deep features from different frequency band feature maps through a double-branch feature extraction structure to obtain low-frequency deep feature maps and high-frequency deep feature maps; subsequently, cross-domain fusion is performed on the low-frequency deep feature maps and the high-frequency deep feature maps by using a deformable cross-attention mechanism to obtain high-low frequency fusion features; the above frequency band decomposition, double-branch extraction and cross-domain fusion operations are repeated along the network depth direction, and the deep feature maps are obtained by combining residual connections.

[0053] The image reconstruction stage is configured to perform convolution, upsampling and overfitting suppression on the deep feature maps to generate a high-resolution reconstructed image of a target scale.

[0054] The optimization module is configured to calculate the difference between the high-resolution reconstructed image and the real high-resolution image by using a pixel-level loss function, optimize the network parameters based on the difference, and obtain an optimized single image super-resolution network to complete real-time single image super-resolution reconstruction.

[0055] The single image super-resolution reconstruction method based on wavelet transform and cross-domain feature fusion has an end-to-end trainable fusion network for single image super-resolution reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 A flowchart of the single image super-resolution reconstruction method based on wavelet transform and cross-domain feature fusion in an embodiment of the present application is shown in FIG. 1.

[0057] Figure 2 A schematic diagram of the overall framework of the single image super-resolution reconstruction method based on wavelet transform and cross-domain feature fusion in an embodiment of the present application is shown in FIG. 2.

[0058] Figure 3 A schematic diagram of the deformable cross-attention mechanism framework in an embodiment of the present application is shown in FIG. 3.

[0059] Figure 4 A schematic diagram of the offset network and mask network framework in an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the accompanying drawings.

[0061] In one embodiment, as shown in FIG. 1, a single image super-resolution reconstruction method based on wavelet transform and cross-domain feature fusion includes the following steps: Figure 1

[0062] S100: Obtain a low-resolution RGB image and construct a single image super-resolution network, which includes a shallow feature extraction module, a deep feature extraction module, and a high-quality image reconstruction stage in sequence, wherein the deep feature extraction module includes a plurality of wavelet feature mixing modules.

[0063] ​S200: input the low-resolution RGB image into the shallow feature extraction module, use the convolution structure to convert the input image features, map it from the RGB space to the high-dimensional feature space, and obtain the low-resolution shallow feature map;

[0064] S300: input the shallow feature map into the wavelet feature mixing module in the deep feature extraction module, first up-sample and discrete wavelet transform the shallow feature map to obtain a multi-band feature map; then use a double-branch feature extraction structure to extract deep features from different frequency band feature maps to obtain low-frequency deep feature maps and high-frequency deep feature maps; then use a deformable cross-attention mechanism to fuse the low-frequency deep feature maps and the high-frequency deep feature maps to obtain high-low frequency fusion features; repeat the above frequency band decomposition, double-branch extraction and cross-domain fusion operations along the network depth direction, and combine the residual connection to obtain a deep feature map;

[0065] S400: perform convolution, up-sampling and overfitting suppression processing on the deep feature map to generate a high-resolution reconstructed image of a target scale;

[0066] S500: use a pixel-level loss function to calculate the difference between the high-resolution reconstructed image and the real high-resolution image, optimize the network parameters based on the difference, and obtain an optimized single image super-resolution network to complete real-time single image super-resolution reconstruction.

[0067] Specifically, the present application proposes a single image super-resolution reconstruction method based on wavelet transform and cross-domain feature fusion, aiming to solve the problem of insufficient utilization of multi-frequency domain information in the current field. The method effectively fuses high-frequency edge details and low-frequency global information through wavelet feature mixing and multi-layer progressive feature extraction, significantly improving the performance of the reconstructed image in terms of texture clarity and structural integrity. In addition, the present application introduces a deformable cross-attention (DCA) mechanism to adaptively align and fuse the low-frequency and high-frequency features generated by wavelet transform, thereby fully exploiting valuable information in the source image in multiple frequency domains, reducing the computational complexity while suppressing the negative impact of high-frequency redundant information on the reconstruction quality, and enhancing the detail expression capability. To further optimize feature extraction, the present application also designs a double-branch architecture that combines the advantages of CNN in capturing high-frequency edge details and Transformer in modeling low-frequency global information, thereby improving the saliency of high-frequency details and the coherence of low-frequency structures in the reconstructed image, and ultimately achieving high-quality super-resolution image reconstruction.

[0068] In one embodiment, the convolution structure in S200 is a hierarchical convolution structure composed of a 7x7 convolution layer and a 3x3 convolution layer; wherein the 7x7 convolution layer is used to capture global context information of the image, and the 3x3 convolution layer is used to locally refine the captured features, and finally generate a low-resolution shallow feature map, which is specifically:

[0069] F S = H SF (I LR )

[0070] where I LR ∈ R 3×h×w is the low-resolution image, h and w are the height and width of the image, respectively, F S is the low-resolution shallow feature map, H SF () represents a shallow feature extraction operation.

[0071] In one embodiment, the up-sampling factor in S300 is two; the discrete wavelet transform decomposes the up-sampled feature map into a low-low frequency feature map LL, a low-high frequency feature map LH, a high-low frequency feature map HL, and a high-high frequency feature map HH, and the sizes of the decomposed feature maps remain consistent with the input shallow feature map.

[0072] Further, the present application designs a double-branch structure composed of an edge feature extraction block (EFEB) based on a CNN architecture and a smooth feature extraction block (SFEB) based on a Transformer architecture, which are respectively used to effectively extract high-frequency and low-frequency features. In addition, the present application also proposes a deformable cross-attention mechanism (DCA), which adaptively aligns and fuses multi-frequency features, enhancing the interaction between spatial and frequency domain components. The above modules together constitute a wavelet feature mixing (WFM) module.

[0073] Given that Transformers are better at capturing low-frequency features, when constructing SFEB, the present application takes advantage of the powerful low-frequency information extraction capability and inherent long-range receptive field of Transformers to efficiently capture LL frequency domain data F LL , as shown in Figure 2 SFEB is composed of a standard Swin transformer module and an overlapping cross-attention module, in addition to convolutional layers, these components all contain an attention module and a feed-forward network.

[0074] Given that CNNs are better at capturing high-frequency features, when constructing EFEB, the present application makes full use of the powerful local feature extraction capability of CNNs to capture high-frequency information, thereby efficiently extracting features F LH , F HL , and F HH from the LH, HL, and HH frequency domains in wavelet changes, respectively, as shown inFigure 2 EFEB enhances the standard convolution block by introducing two convolution layers with normalization and channel attention mechanism based on average pooling. Since the input of EFEB contains three high-frequency bands, the channel attention mechanism effectively alleviates the feature confusion problem.

[0075] In an embodiment, the double-branch feature extraction structure in S300 includes a smooth feature extraction block SFEB based on a Transformer architecture and an edge feature extraction block EFEB based on a CNN architecture; SFEB is composed of a convolution layer, a window multi-head self-attention module, a Swin Transformer module and an overlapping cross-attention module OCA connected in sequence, SFEB is used to process the low-low frequency feature map LL to obtain a low-frequency depth feature map, and its output expression is:

[0076]

[0077] wherein, is the input feature, H conv (·) represents a convolution operation, H win (·) represents a window multi-head self-attention operation, H swin (·) represents a Swin Transformer processing operation, H OCA (·) represents an overlapping cross-attention operation, is the low-frequency depth feature map;

[0078] The window multi-head self-attention module divides the input feature map into non-overlapping local windows of MxM size, and independently performs self-attention calculation on each window, and the total number of windows is M is the window side length; the overlapping cross-attention module OCA adopts a sliding window strategy, and expands the window into an overlapping window of MoxMo size, and Mo is the overlapping window side length.

[0079] EFEB includes two convolution layers with normalization and channel attention mechanism based on average pooling, and EFEB is used to process the spliced features of the low-high frequency feature map LH, the high-low frequency feature map HL and the high-high frequency feature map HH to obtain a high-frequency depth feature map, and its output expression is:

[0080]

[0081] wherein, represents the input feature of EFEB, H Li (·) represents a linear transformation operation, H and W are the height and width of the feature map respectively, sigmoid(·) represents an activation function, is the high-frequency depth feature map.

[0082] Further, low-frequency components mainly capture global content information, while high-frequency components encode fine-grained edge details. Fortunately, wavelet transform provides an efficient and lossless frequency decomposition technique, which is widely used in image analysis. However, in wavelet transform, the number of high-frequency channels is three times that of low-frequency channels, and a considerable portion of high-frequency pixels has low values. Blindly using all high-frequency channels will lead to excessive computational overhead and may degrade the performance of SR. Specifically, using a Transformer directly to implement cross-domain attention mechanisms will introduce a spatial complexity of 3N Q N K C, where N Q and N K represent the number of queries and keys, which are usually proportional to the feature map size HW, resulting in a complexity of approximately quadratic with respect to the spatial dimension.

[0083] To alleviate this problem, the present application proposes a new attention module named DCA, which can adaptively and spatially flexibly fuse multiple wavelet subbands, as shown in FIG. 1. DCA aims to adaptively distinguish the relative importance of low-frequency and high-frequency components to image SR, effectively preserving high-frequency details while reducing the spatial complexity to 3NC, where N represents the number of pixels that need self-attention adaptively selected. The present application designs an offset network and a mask network to facilitate the deformable attention mechanism, as shown in FIG. 2. Figure 3 Figure 4

[0084] In one embodiment, a deformable cross-attention mechanism is adopted in S300 to perform cross-domain fusion of low-frequency deep feature maps and high-frequency deep feature maps, obtaining high-low frequency fusion features, including:

[0085] A low-frequency offset matrix and a high-frequency offset matrix are calculated by a low-frequency offset network and a high-frequency offset network, respectively, and a deformable attention mask is calculated by a mask network on the spliced features of the low-frequency deep feature maps and the high-frequency deep feature maps;

[0086] Pixel-level optical flow warping operations are performed on the low-frequency deep feature maps and the high-frequency deep feature maps based on the low-frequency offset matrix and the high-frequency offset matrix, respectively, a query matrix Q and a value matrix V are constructed based on the low-frequency deep feature maps after optical flow warping and the deformable attention mask, a key matrix K is constructed based on the high-frequency deep feature maps after optical flow warping and the deformable attention mask, and a 3x3 convolution is used to process the part that does not perform attention calculation.

[0087] In one embodiment, the calculation of the low-frequency offset matrix, the high-frequency offset matrix, and the deformable attention mask is specifically as follows:

[0088] ​​

[0089]

[0090] θ mask =H mask (concat(F L ,F H ))

[0091] where F L and F H are low-frequency and high-frequency depth feature maps respectively, and represent low-frequency and high-frequency offset networks respectively, H mask (·) denotes a mask network, concat(·,·) denotes concatenation along the channel dimension, θ mask is a deformable attention mask, and are low-frequency and high-frequency offset matrices respectively;

[0092] The deformable attention mask output by the mask network is used to screen out K windows for sparse attention calculation, and the remaining n-K windows are processed by convolution, where n is the total number of windows of the feature map, and the calculation formula is ws is the window size; the attention calculation formula is as follows:

[0093]

[0094]

[0095] Q = θ mask × F L ’ × W q , K = θ mask × F H ’ × W k

[0096] V = θ mask × F L ’ × W v , W S = (1- θ mask ) × F L ’ × W v

[0097]

[0098] where Flowwarp(·) represents an optical flow warping operation, W q , W k and W v are query matrix, key matrix and value matrix respectively, d represents the dimension of the query / key, B represents the learnable relative position encoding, F LF is the low-frequency depth feature map after optical flow distortion, F H F is the high-frequency depth feature map after optical flow distortion, is the output of deformable cross-attention. It can be seen that in the attention mechanism, the key matrix is derived from the high-frequency information, while the query matrix and the value matrix are derived from the low-frequency component.

[0099] In one embodiment, the above frequency band decomposition, double-branch extraction and cross-domain fusion operations are repeated along the network depth direction in S300, combined with residual connection to obtain a depth feature map, specifically:

[0100]

[0101]

[0102] wherein, represents the target feature map output by the last deformable cross-attention module, represents the target feature map output by the i-th deformable cross-attention module, represents the operation of the i-th wavelet feature mixing module WFM, F R represents the depth feature map, represents the input of the first WFM module.

[0103] In one embodiment, the high-quality image reconstruction module in S400 is composed of a convolutional layer, an upsampling operation and a dropout layer, and S400 is specifically:

[0104] I SR =H HQIR (F R )

[0105] wherein H HQIR (·) represents a high-quality image reconstruction operation, I SR represents a high-resolution reconstructed image.

[0106] Specifically, the addition of the dropout layer can alleviate the overfitting problem of the model and improve its generalization ability.

[0107] In one embodiment, the pixel-level loss function in S500 is specifically:

[0108]

[0109] wherein i represents the image index, N represents the number of training pairs, I SR is the output obtained by taking I LR as the input of the network, and I HR is the corresponding true value, i.e. the high-quality image.

[0110] The present application proposes a novel wavelet transform and cross-domain feature fusion method which is currently lacking in the field of image super-resolution. At the same time, existing super-resolution reconstruction methods mainly focus on improving the basic architecture design of CNN or Transformer, but such methods often ignore the key role of frequency domain information, especially high-frequency components, in detail reconstruction. Although some recent studies have attempted to introduce frequency domain transformation methods to assist super-resolution reconstruction, these methods still have obvious limitations: they usually use the same network architecture to process features of different frequency bands, without fully considering the feature differences between low-frequency global information and high-frequency local details; In the cross-frequency feature fusion stage, it mostly relies on simple operations, and this coarse-grained fusion method not only fails to fully exploit the complementary value of different frequency band features, but also may introduce a large amount of redundant information. Specifically, the current frequency-based super-resolution reconstruction method has the following key problems: single frequency domain decomposition method, feature extraction network homogenization, and simple feature fusion mechanism, which restricts the further improvement of reconstruction effect. Based on the above problems, the present application proposes a single image super-resolution reconstruction method based on wavelet transform and cross-domain feature fusion, and the main advantages of the present application are as follows:

[0111] (1) The fusion network of the present application is an end-to-end trainable network that can learn feature extraction, feature fusion and image reconstruction through training. This method realizes effective fusion of high-frequency edge details and low-frequency global information through wavelet feature mixing and multi-layer progressive feature extraction, significantly improves the texture clarity and structural integrity of the reconstructed image, and obtains high-quality super-resolution images.

[0112] (2) The present application realizes multi-frequency domain feature extraction and deformable cross-attention mechanism guided by wavelet transform, and performs high-frequency and low-frequency feature fusion based on the adaptive weight of frequency domain decomposition, fully excavates the internal relationship between edge details and global information in the source image, and effectively avoids the interference of redundant information, so that the reconstructed image retains clear texture details and structural integrity.

[0113] (3) The present application designs a double-branch structure, including a CNN-based edge feature extraction block and a Transformer-based smooth feature extraction block, which respectively highlights the extraction ability of high-frequency edge information and low-frequency global information, enhances the saliency of high-frequency details in the reconstructed image, and thus improves the contrast and visual clarity of the overall image.

[0114] (4) The present application designs an optimization function based on L1 pixel loss to guide network training to improve the sensitivity to multi-frequency domain features in the source image and avoid the loss of high-frequency details. At the same time, by reasonably allocating the fusion weights of low-frequency and high-frequency features, it realizes the balanced reconstruction of intensity information and texture details, thereby obtaining excellent super-resolution effect.

[0115] In one embodiment, a single image super-resolution reconstruction system based on wavelet transform and cross-domain feature fusion is also provided, comprising:

[0116] A single image super-resolution network module is configured to obtain a low-resolution RGB image, and construct a single image super-resolution network, which sequentially comprises a shallow feature extraction module, a deep feature extraction module, and a high-quality image reconstruction stage. The deep feature extraction module comprises a plurality of wavelet feature mixing modules.

[0117] The shallow feature extraction module is configured to convert the input low-resolution RGB image into a high-dimensional feature space using a convolution structure, thereby mapping it from an RGB space to a high-dimensional feature space, and obtaining a low-resolution shallow feature map.

[0118] The deep feature extraction module is configured to upsample and perform discrete wavelet transform on the shallow feature map to obtain a multi-band feature map. Then, the deep feature extraction module is configured to perform deep feature extraction on different frequency band feature maps through a double-branch feature extraction structure to obtain low-frequency deep feature maps and high-frequency deep feature maps. Subsequently, the deep feature extraction module is configured to perform cross-domain fusion on the low-frequency deep feature maps and the high-frequency deep feature maps using a deformable cross-attention mechanism to obtain high-low frequency fusion features. The above frequency band decomposition, double-branch extraction, and cross-domain fusion operations are repeated along the network depth direction, and the deep feature maps are obtained by combining the residual connection.

[0119] The image reconstruction stage is configured to perform convolution, upsampling, and overfitting suppression on the deep feature maps to generate a high-resolution reconstructed image of a target scale.

[0120] The optimization module is configured to calculate the difference between the high-resolution reconstructed image and the real high-resolution image using a pixel-level loss function, optimize the network parameters based on the difference, and obtain an optimized single image super-resolution network to complete real-time single image super-resolution reconstruction.

[0121] The specific limitations of the single image super-resolution reconstruction system based on wavelet transform and cross-domain feature fusion can be referred to the limitations of the single image super-resolution reconstruction method based on wavelet transform and cross-domain feature fusion in the foregoing, which will not be repeated here. Each module in the single image super-resolution reconstruction system based on wavelet transform and cross-domain feature fusion can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0122] The above describes in detail the method and system for single image super-resolution reconstruction based on wavelet transform and cross-domain feature fusion provided by the present application. The principles and implementation manners of the present application are described by using specific examples in this paper, and the above description of the examples is only used to help understand the core idea of the present application. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A single-image super-resolution reconstruction method based on wavelet transform and cross-domain feature fusion, characterized in that, The method includes the following steps: S100: Acquire low-resolution RGB images and construct a single-image super-resolution network. This network sequentially includes a shallow feature extraction module, a deep feature extraction module, and a high-quality image reconstruction stage. The deep feature extraction module includes multiple wavelet feature mixing modules. S200: Input the low-resolution RGB image into the shallow feature extraction module, use a convolutional structure to perform feature transformation on the input image, and map it from the RGB space to a high-dimensional feature space to obtain a low-resolution shallow feature map; S300: The shallow feature map is input into the wavelet feature mixing module in the deep feature extraction module. First, the shallow feature map is upsampled and subjected to discrete wavelet transform to obtain a multi-band feature map. Then, a dual-branch feature extraction structure is used to extract deep features from the feature maps of different frequency bands to obtain low-frequency and high-frequency deep feature maps. Subsequently, a deformable cross-attention mechanism is used to perform cross-domain fusion of the low-frequency and high-frequency deep feature maps to obtain high-low frequency fused features. The above frequency band decomposition, dual-branch extraction and cross-domain fusion operations are repeated along the network depth direction, and the deep feature map is obtained by combining residual connections. S400: Performs convolution, upsampling, and overfitting suppression on the depth feature map to generate a high-resolution reconstructed image at the target scale; S500: It uses a pixel-level loss function to calculate the difference between the high-resolution reconstructed image and the real high-resolution image, optimizes the network parameters based on the difference, and obtains an optimized single-image super-resolution network to complete real-time single-image super-resolution reconstruction.

2. The method according to claim 1, characterized in that, The S200 convolutional structure is a hierarchical convolutional structure, consisting of 7×7 convolutional layers and 3×3 convolutional layers. The 7×7 convolutional layers capture global contextual information of the image, while the 3×3 convolutional layers refine the captured features locally, ultimately generating a low-resolution shallow feature map. Specifically: F S =H SF (I LR ) Among them, I LR ∈R 3×h×w For low-resolution images, h and w are the height and width of the image, respectively, and F S H is a low-resolution shallow feature map. SF () indicates shallow feature extraction operation.

3. The method according to claim 2, characterized in that, In S300, the upsampling factor is twice; the discrete wavelet transform decomposes the upsampled feature map into low-low frequency feature map LL, low-high frequency feature map LH, high-low frequency feature map HL, and high-high frequency feature map HH, and the size of each feature map after decomposition is consistent with that of the input shallow feature map.

4. The method according to claim 3, characterized in that, The dual-branch feature extraction structure in S300 includes the Smooth Feature Extraction Block (SFEB) based on the Transformer architecture and the Edge Feature Extraction Block (EFEB) based on the CNN architecture. SFEB consists of convolutional layers, a window multi-head self-attention module, a Swing Transformer module, and an Overlapping Cross Attention (OCA) module connected sequentially. SFEB is used to process the low-frequency feature map LL to obtain the low-frequency depth feature map, and its output expression is: in, For input features, H conv (·) represents the convolution operation, H win (·) indicates a multi-head self-attention operation for the window, H swin (·) indicates a Swing Transformer processing operation, H OCA (·) indicates an overlapping attention operation. This is a low-frequency depth feature map; The multi-head self-attention module divides the input feature map into M×M non-overlapping local windows, and performs self-attention calculation independently for each window, for a total of M×M windows. M is the side length of the window; the Overlapping Cross Attention (OCA) module uses a sliding window strategy to expand the window into an overlapping window of size Mo×Mo, where Mo is the side length of the overlapping window; EFEB consists of two normalized convolutional layers and a channel attention mechanism based on average pooling. EFEB is used to process the concatenated features of low-frequency feature map LH, high-low frequency feature map HL, and high-frequency feature map HH to obtain a high-frequency depth feature map. Its output expression is: in, H represents the input features of EFEB. Li (·) denotes a linear transformation operation, H and W are the height and width of the feature map, respectively, and sigmoid(·) represents the activation function. This is a high-frequency depth feature map.

5. The method according to claim 4, characterized in that, In S300, a deformable cross-attention mechanism is used to perform cross-domain fusion of low-frequency and high-frequency depth feature maps, resulting in high-low frequency fused features, including: The low-frequency offset matrix and the high-frequency offset matrix are calculated by the low-frequency offset network and the high-frequency offset network respectively. The deformable attention mask is obtained by calculating the concatenated features of the low-frequency depth feature map and the high-frequency depth feature map through the mask network. Pixel-level optical flow warping operations are performed on the low-frequency depth feature map and the high-frequency depth feature map based on the low-frequency offset matrix and the high-frequency offset matrix, respectively. The query matrix Q and the value matrix V are constructed using the optically warped low-frequency depth feature map and the deformable attention mask. The key matrix K is constructed using the optically warped high-frequency depth feature map and the deformable attention mask. The part for which attention calculation was not performed is processed by 3×3 convolution.

6. The method according to claim 5, characterized in that, The calculation of the low-frequency offset matrix, the high-frequency offset matrix, and the deformable attention mask is as follows: θ mask =H mask (concat(F L ,F H )) Among them, F L and F H These are low-frequency depth feature maps and high-frequency depth feature maps, respectively. and H represents the low-frequency and high-frequency offset networks, respectively. mask (·) represents a mask network, concat(·,·) represents concatenation along the channel dimension, θ mask For deformable attention masks, and These are the low-frequency offset matrix and the high-frequency offset matrix, respectively; The deformable attention mask output by the masked network is used to select K windows for sparse attention computation. The remaining nK windows are then convolved, where n is the total number of windows in the feature map, calculated using the following formula: ws is the window size; the attention calculation formula is as follows: Q=θ mask ×F L '×W q ,K=θ mask ×F H '×W k V=θ mask ×F L '×W v ,V S =(1-θ mask )×F L '×W v Where Flowarp(·) represents the optical flow warping operation, W q W k and W v These are the query matrix, key matrix, and value matrix, respectively. d represents the query / key dimension, B represents the learnable relative position encoding, and F... L 'This is a low-frequency depth feature map after optical flow distortion, F' H 'This is a high-frequency depth feature map after optical flow distortion.' The output is a deformable cross-attention function.

7. The method according to claim 6, characterized in that, In S300, the above frequency band decomposition, bi-branch extraction, and cross-domain fusion operations are repeated along the network depth direction. Combined with residual connections, a deep feature map is obtained, specifically: in, This represents the target feature map output by the last deformable cross-attention module. This represents the target feature map output by the i-th deformable cross-attention module. F represents the operation of the i-th wavelet feature mixing module (WFM). R Represents a depth feature map. This represents the input of the first WFM module.

8. The method according to claim 7, characterized in that, The high-quality image reconstruction module in S400 consists of convolutional layers, upsampling operations, and dropout layers. Specifically, S400 is as follows: I SR =H HQIR (F R ) Where H HQIR (·) indicates a high-quality image reconstruction operation, I SR This indicates a high-resolution reconstructed image.

9. The method according to claim 8, characterized in that, The pixel-level loss function in S500 is as follows: Where i represents the image index, N represents the number of training pairs, and I SR It is by I LR The output obtained as input to the network, and I HR That corresponds to the true value, i.e., a high-quality image.

10. A single-image super-resolution reconstruction system based on wavelet transform and cross-domain feature fusion, characterized in that, include: A single-image super-resolution network module is used to acquire low-resolution RGB images and construct a single-image super-resolution network. This network sequentially includes a shallow feature extraction module, a deep feature extraction module, and a high-quality image reconstruction stage. The deep feature extraction module includes multiple wavelet feature mixing modules. The shallow feature extraction module is used to perform feature transformation on the input low-resolution RGB image using a convolutional structure, mapping it from the RGB space to a high-dimensional feature space to obtain a low-resolution shallow feature map. The deep feature extraction module is used to upsample and perform discrete wavelet transform on the shallow feature map to obtain a multi-band feature map. Then, a dual-branch feature extraction structure is used to extract deep features from the feature maps of different frequency bands to obtain low-frequency and high-frequency deep feature maps. Subsequently, a deformable cross-attention mechanism is used to perform cross-domain fusion of the low-frequency and high-frequency deep feature maps to obtain high-low frequency fused features. The above frequency band decomposition, dual-branch extraction and cross-domain fusion operations are repeated along the network depth direction, and combined with residual connections to obtain the deep feature map. In the image reconstruction stage, it is used to perform convolution, upsampling and overfitting suppression on the depth feature map to generate a high-resolution reconstructed image at the target scale; The optimization module is used to calculate the difference between the high-resolution reconstructed image and the real high-resolution image using a pixel-level loss function. Based on this difference, the network parameters are optimized to obtain an optimized single-image super-resolution network that completes real-time single-image super-resolution reconstruction.

Images