Image restoration method using space channel shuffling mechanism and high and low frequency up-sampling
By introducing a spatial channel shuffling mechanism and high- and low-frequency upsampling techniques, the problems of insufficient detail recovery and edge discontinuity in complex areas in existing image restoration methods are solved, achieving higher quality image restoration results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-13
- Publication Date
- 2026-04-14
AI Technical Summary
Existing image restoration methods often face problems of insufficient detail restoration and discontinuous edges when dealing with large-scale missing areas and complex regions, especially in the restoration of complex backgrounds and textures.
An image restoration method employing a spatial channel shuffling mechanism and high- and low-frequency upsampling is proposed. By introducing spatial channel shuffling attention and high- and low-frequency fusion upsampling technology, the detail restoration capability and edge continuity of the image restoration are improved.
It significantly improves restoration quality, enhances detail recovery, edge continuity, and image structure consistency, thereby improving restoration results.
Smart Images
Figure CN121860893A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image restoration technology, specifically relating to an image restoration method that utilizes a spatial channel shuffling mechanism and high and low frequency upsampling. Background Technology
[0002] Image restoration is an important research area in image processing, with wide applications covering historical artifact restoration, medical image restoration, photographic restoration, object removal, and many other fields. Its goal is to fill in missing or damaged areas in an image, restoring the missing content so that the restored image visually resembles the original as closely as possible. Figure 1 To.
[0003] Traditional image restoration methods are mostly based on techniques such as pixel interpolation and texture synthesis. These methods fill in the gaps by utilizing existing information in the image, and can achieve good results when dealing with simple image defects. However, as the missing area increases or the image background becomes more complex, these traditional methods cannot fully restore details, especially in the restoration of complex textures and image structures, where the restoration effect falls far short of the ideal level.
[0004] In recent years, the application of deep learning, especially generative adversarial networks (GANs) and convolutional neural networks (CNNs), in image inpainting has greatly advanced this technology. GANs, through adversarial training between the generator and discriminator, have effectively improved the realism of inpainting results, particularly in detail restoration and image naturalness. Despite these improvements in inpainting quality, limitations remain in the restoration of complex regions, especially in the restoration of structural boundaries, details, and textures. For example, when dealing with complex backgrounds and large-scale occlusion, existing methods still fall short of ideal results in detail restoration and edge continuity. Furthermore, problems such as detail loss and blurred edges persist when facing more complex inpainting tasks.
[0005] Therefore, how to simultaneously restore details, maintain edge continuity, and improve restoration accuracy in the face of large-scale occlusion and complex backgrounds remains a pressing technical challenge in the field of image restoration. Summary of the Invention
[0006] Existing image restoration methods often suffer from insufficient detail restoration and edge discontinuities when dealing with large-scale missing areas and complex regions. The process of handling high-frequency details and structural information in images by existing technologies results in poor performance in restoring complex backgrounds and textures. This is particularly true in tasks such as the restoration of historical artifacts, where it is difficult to ensure detail restoration and structural consistency. To address these issues, this invention provides an image restoration method utilizing a spatial channel shuffling mechanism and high- and low-frequency upsampling. By introducing spatial channel shuffling attention and high- and low-frequency fusion upsampling techniques, it effectively improves the detail restoration capability and edge continuity of image restoration, overcoming the shortcomings of existing methods in complex image restoration.
[0007] The technical solution of this invention is implemented as follows: an image inpainting method utilizing a spatial channel shuffling mechanism and high- and low-frequency upsampling, comprising the following steps: S1. Obtain the input image to be repaired and the corresponding mask, wherein the mask is used to indicate the missing area in the input image; S2. Input the input image into the convolutional feature extraction module to obtain the initial feature map; S3. Input the initial feature map into the encoding end network composed of multiple stacked Orea modules. The Orea module is a feedforward network-spatial channel shuffle attention-feedforward network structure, and the encoding end performs step-by-step downsampling through the downsampling module to obtain multi-scale features. S4. Input the features from the encoding end into the decoding end network. The decoding end includes a high-low frequency fusion upsampling module to upsample the features and complete the separation, enhancement and fusion of high-frequency features and low-frequency features during the upsampling process. S5. The features from the decoding end are spliced and fused with the corresponding scale features from the encoding end, and the restored image is output by the reconstruction module.
[0008] Furthermore, the spatial channel shuffling attention in S3 includes window self-attention and channel shuffling operation. Window self-attention obtains attention output by calculating query matrix Q, key matrix K, and value matrix V, and performs residual fusion with input features.
[0009] Furthermore, the high-low frequency fusion upsampling module in S4 includes: performing a two-dimensional Fourier transform in the frequency domain after upsampling the features to separate the high-frequency components and low-frequency components, and using Laplace filtering to extract high-frequency features and Gaussian smoothing to extract low-frequency features in the spatial domain.
[0010] Furthermore, in S4, the high- and low-frequency fusion upsampling module is set with learnable dynamic weights to perform weighted fusion of high-frequency features and low-frequency features to obtain upsampled output features.
[0011] This invention offers the following advantages: by employing a network structure combining spatial channel shuffling attention and high-low frequency fusion upsampling techniques for image inpainting, the inpainting effect is significantly improved. Compared to traditional image inpainting methods, this invention demonstrates clear advantages in terms of inpainting quality, detail restoration, edge continuity, and image structure consistency. Attached Figure Description
[0012] Appendix Figure 1 This is a flowchart of the overall network of the present invention.
[0013] Appendix Figure 2 This is the overall network design diagram of the present invention.
[0014] Appendix Figure 3 This is a design diagram of the spatial channel shuffling attention module of the present invention.
[0015] Appendix Figure 4 This is the high-frequency fusion upsampling design diagram of the present invention. Detailed Implementation
[0016] An image inpainting method utilizing a spatial channel shuffling mechanism and high / low frequency upsampling includes the following steps: Step 1: Prepare the dataset. First, select a dataset suitable for the image restoration task. The dataset should include diverse image samples and their corresponding missing regions. These images can come from natural images and images from the restoration of historical artifacts, ensuring the diversity and representativeness of the dataset.
[0017] Step 2: Data Preprocessing. Since some raw images cannot be directly input into the network for training, image preprocessing is necessary. This includes image cropping, scaling, and normalization to ensure the image data meets the input requirements of the network structure.
[0018] Step 3: Design the network structure. The encoding end of the network consists of convolutional layers and multi-level Orea modules, and downsampling is achieved step by step through downsampling; the decoding end introduces an HLF-Upsampling module and merges it with the features from the encoding end to restore the structural and texture details of the missing regions.
[0019] Step 4: Use PyCharm programming software to train the network within the PyTorch deep learning framework. Train the improved network in this environment.
[0020] Step 5: Input the dataset obtained in Step 2 into the deep learning network structure set in Step 4 and train it. The trained network structure is the final network structure model.
[0021] Step 6: Input the test set into the network model to obtain the segmentation results.
[0022] The present invention will be further described below with reference to the accompanying drawings, and the present invention includes, but is not limited to, the following embodiments. The specific implementation process is as follows: Step 1: Download the dataset. A diverse dataset was used, including images of Dunhuang artifacts, images of various artistic styles, and facial images, to facilitate image restoration in different application scenarios. The Dunhuang artifact image dataset contains images of damaged artifacts from the Dunhuang site, which present unique challenges for evaluating restoration techniques. The artistic style image dataset contains images of various artworks with rich texture and structural features. The facial image dataset provides high-resolution facial images, primarily used to evaluate the effectiveness of facial feature restoration.
[0023] Step 2: Data Preprocessing. Data preprocessing is a crucial step in ensuring that images from different datasets can be input into the network for training. First, all images need to be resized, scaling them to a fixed size (256×256 pixels) to fit the input requirements of the network structure. To enhance the model's generalization ability, data augmentation techniques such as rotation, cropping, and mirroring are also used to improve the network's robustness to restoration tasks in different scenarios.
[0024] Step 3: Train the network structure using a deep learning network model.
[0025] Step 3-1: Input Image Processing and Feature Extraction. Input Image After convolution operation Extracting preliminary features: ,in Indicates the input image Preliminary feature representation.
[0026] Step 3-2: Channel attention branch of the SCSA module. For the feature map... Perform channel dimension calibration: unfold the feature map in spatial dimension as (1) The channel weights are obtained through two fully connected layers (MLP) and then normalized using a sigmoid function, resulting in the following channel attention map: (2) The final output is (3) in For linear transformation parameters, It is the ReLU activation function. For the Sigmoid function, This indicates element-wise multiplication.
[0027] Step 3-3: Window QKV self-attention, channel shuffling, and spatial attention branches in the SCSA module. By window size Divide the data into several windows, and within each window, obtain the query, key, and value matrix through linear mapping: (4) And calculate the multi-head self-attention output within the window. , (5) The attention output is mapped back to the feature space and added to the residual of the input to obtain... (6) Then on Perform a channel shuffle to enhance cross-group channel interaction, resulting in Then, a spatial attention branch (depthseparable 7×7 convolution) is introduced to generate spatial weights. And perform gating: (7) Where d is the dimension of each attention head. , These are learnable parameters.
[0028] Steps 3-4: High-low frequency fusion upsampling (HLF-Upsampling). For features... First, perform an upsampling operation (PixelShuffle after convolution) to obtain... Then, high and low frequency information is separated and fused in the frequency and spatial domains. Frequency domain separation uses a two-dimensional FFT: (8) (9) (10) (11) In the spatial domain, high frequencies are extracted using the Laplacian operator, and low frequencies are extracted using Gaussian blur. (12) (13) The high and low frequencies in the frequency and spatial domains are enhanced by convolution, and an edge enhancement module is introduced for the high frequencies to obtain... Low frequency is obtained Finally, learnable dynamic weights are used for fusion: (14) in, For dynamic weights , These are learnable parameters.
[0029] Steps 3-5: Final Image Restoration: Restoration Feature Maps Obtained Through the Above Steps After final upsampling and convolution operations, the missing regions of the image are restored, and the repaired image is output. (15) Step 4: Use PyCharm programming software. The software environment is Python 3.8. The hardware environment is: 13th Gen Intel(R) Core(TM) i9-13900K 3.00 GHz processor, Win10 64-bit, and NVIDIA GeForce RTX 3090 system. Step 5: Input the dataset obtained in Step 2 into the network structure set in Step 4 and train it. The trained network structure is the final network structure model.
[0030] Step 6: Input the test set for testing. Input the test set portion from Step 2 into the trained deep learning model, train the model on the test set, obtain the test set results, and conduct experimental comparisons.
[0031] Table 1 Comparison of metrics on the CeleA-HQ dataset
[0032] Table 2 Comparison of metrics on the Dunhuang dataset
[0033] The experimental results are shown in Tables 1 and 2. The comparison of the results shows that the network incorporating spatial channel shuffling attention and high-low frequency fusion upsampling mechanisms performs better overall in terms of evaluation metrics such as PSNR, SSIM, L1, and LPIPS. Specifically, PSNR measures the pixel-level error between the restored image and the reference image; a higher value indicates a smaller reconstruction error. SSIM measures the consistency of brightness, contrast, and structural information between the two images; a higher value indicates greater structural similarity. L1 represents the average level of absolute pixel error to reflect the overall degree of deviation; a lower value indicates smaller pixel deviation. LPIPS measures perceptual similarity based on deep features to reflect differences in subjective human perception; a lower value indicates that the perceived quality is closer to reality. The comprehensive improvement in these metrics indicates that the network has achieved improvements in pixel fidelity, structural continuity, and perceptual realism.
Claims
1. An image inpainting method utilizing a spatial channel shuffling mechanism and high / low frequency upsampling, characterized in that, Includes the following steps: S1. Obtain the input image to be repaired and the corresponding mask, wherein the mask is used to indicate the missing area in the input image; S2. Input the input image into the convolutional feature extraction module to obtain the initial feature map; S3. Input the initial feature map into an encoding network composed of multiple stacked Orea modules. The Orea module is a feedforward network-spatial channel shuffle attention-feedforward network structure, and the encoding end performs step-by-step downsampling through a downsampling module to obtain multi-scale features. S4. Input the features from the encoding end into the decoding end network. The decoding end includes a high-low frequency fusion upsampling module to upsample the features and complete the separation, enhancement and fusion of high-frequency features and low-frequency features during the upsampling process. S5. The features from the decoding end are spliced and fused with the corresponding scale features from the encoding end, and the restored image is output by the reconstruction module.
2. The image inpainting method using spatial channel shuffling mechanism and high / low frequency upsampling according to claim 1, characterized in that: The spatial channel shuffling attention in S3 includes window self-attention and channel shuffling operation. Window self-attention obtains attention output by calculating query matrix Q, key matrix K, and value matrix V, and performs residual fusion with input features.
3. The image inpainting method using spatial channel shuffling mechanism and high / low frequency upsampling according to claim 1, characterized in that: The high-low frequency fusion upsampling module in S4 includes: performing a two-dimensional Fourier transform in the frequency domain after upsampling the features to separate the high-frequency components and low-frequency components, and using Laplace filtering to extract high-frequency features and Gaussian smoothing to extract low-frequency features in the spatial domain.
4. The image inpainting method using spatial channel shuffling mechanism and high / low frequency upsampling according to claim 1, characterized in that: The high- and low-frequency fusion upsampling module in S4 is equipped with learnable dynamic weights to perform weighted fusion of high-frequency features and low-frequency features to obtain upsampled output features.