Space and frequency domain data enhancement method for image super-resolution

Abstract

The invention provides a space and frequency domain data enhancement method for image super-resolution, and relates to the field of image processing. In order to solve the problems that in the prior art, picture information utilization is insufficient, and spatial correlation is prone to being damaged by a high-dimensional task method, bit plane transformation is carried out on a low-resolution image by transforming an image bit plane and introducing frequency domain fusion, and multiple reconstructed bit planes are obtained; and performing bit plane fusion on the plurality of bit planes of the high-resolution image and the plurality of reconstructed bit planes of the low-resolution image to obtain an enhanced low-resolution image fused with frequency domain information, and inputting the enhanced low-resolution image into the super-resolution network to generate a corresponding reconstructed image. According to the method, association between low-resolution underlying features and high-resolution frequency domain information is established. On the basis of protecting image spatial correlation, bit-level hidden information of a low-resolution image is mined and deeply fused with frequency and amplitude characteristics of a high-resolution image, so that texture details are greatly enriched.

Description

Technical Field

[0001] This invention belongs to the field of image processing and relates to a method for spatial and frequency domain data enhancement for image super-resolution. Background Technology

[0002] Image super-resolution (SR) is a key technology dedicated to reconstructing and recovering high-quality, high-resolution (HR) images from low-resolution (LR) observations. With the deep application of convolutional neural networks in image processing, deep learning-based SR schemes have achieved breakthroughs in network architecture optimization and training strategy iteration.

[0003] To achieve ideal performance metrics for deep learning models, a large-scale, high-quality image dataset is typically required. Current practices include artificially synthesizing large datasets using specific degradation models or obtaining real samples through time-consuming field collection. When the number of samples is limited, researchers commonly employ data augmentation (DA) strategies, such as performing geometric transformations on images like random cropping, scaling, and flipping, or adjusting the RGB channels for color. For certain high-dimensional tasks, new samples are also derived by modifying pixels or feature values.

[0004] Current technical approaches still have many limitations in practical applications. First, the degradation patterns of synthetic datasets are often out of sync with real-world environments, resulting in insufficient model generalization ability, while constructing massive real-world datasets is extremely expensive. Second, many existing data augmentation methods easily destroy the inherent spatial correlation of images when modifying pixels or features, which negatively impacts low-dimensional visual tasks such as super-resolution that emphasize detail restoration. Furthermore, most existing processing methods remain at the surface pixel or feature level, failing to effectively uncover the hidden bit-level texture and detail information within images, resulting in limited performance improvements for large network models and a loss of data value. Summary of the Invention

[0005] To address the issues of insufficient utilization of image information in existing technologies and the tendency of methods for high-dimensional tasks to destroy spatial correlations, this invention provides a spatial and frequency domain data enhancement method for image super-resolution. By transforming the image bit plane and introducing frequency domain fusion, a correlation is established between low-resolution low-level features and high-resolution frequency domain information.

[0006] This invention provides a spatial and frequency domain data enhancement method for image super-resolution, comprising the following steps:

[0007] High-resolution and low-resolution images of the same shooting scene are analyzed into multiple bit planes; frequency domain information of the high-resolution image is extracted.

[0008] Perform bit-plane transformation on the low-resolution image to obtain multiple reconstructed bit-planes;

[0009] Bit-plane fusion is performed on multiple bit planes of a high-resolution image and multiple reconstructed bit planes of a low-resolution image to obtain an enhanced low-resolution image with fused frequency domain information;

[0010] The low-resolution image is enhanced and input into the super-resolution network to generate the corresponding reconstructed image.

[0011] Furthermore, the specific method for resolving high-resolution and low-resolution images of the same shooting scene into multiple bit planes is as follows:

[0012] The color channels of the high-resolution and low-resolution images are decoupled separately, and each high-resolution and low-resolution image is decomposed into three single-channel images: red, green and blue.

[0013] Each single-channel image is decomposed into planes, which are divided into 8 bit planes, namely the 1st bit plane to the 8th bit plane. The 1st bit plane to the 4th bit plane represent the high-frequency information of the single-channel image, and the 5th bit plane to the 8th bit plane represent the low-frequency information of the single-channel image.

[0014] Furthermore, the specific method for extracting the frequency domain information of the high-resolution image is as follows:

[0015] A fast Fourier transform is performed on all bit planes of the high-resolution image to extract the frequency domain information of the high-resolution image, which includes frequency information and amplitude information.

[0016] Furthermore, the specific method for performing bit-plane transformation on low-resolution images is as follows:

[0017] For each low-resolution image, set the logic for swapping bit planes with the same weight between different channels;

[0018] In each channel, any number of bit planes are randomly selected from the 8 bit planes.

[0019] Based on the defined swapping logic, precise physical position swaps are performed on the selected bit plane to obtain multiple reconstructed bit planes.

[0020] Furthermore, the swapping logic includes pairwise swapping and cyclical permutation between the three channels; pairwise swapping includes swapping the red-green channel, swapping the red-blue channel, and swapping the blue-green channel; cyclical permutation between the three channels includes cyclical permutation of red, green, blue, red in sequence and cyclical permutation of blue, green, red, blue in sequence.

[0021] Furthermore, the specific method for bit-plane fusion is as follows:

[0022] For high-resolution images and reconstructed low-resolution images, several channels can be freely selected from the three independent channels of red, green, and blue;

[0023] For high-resolution images and reconstructed low-resolution images, an arbitrary number of bit planes are randomly selected from the selected channels for frequency information fusion.

[0024] Between each selected channel, the frequency domain information of the high-resolution image bit plane is fused to the reconstructed bit plane corresponding to the low-resolution image. The fusion methods include frequency-based fusion and frequency and amplitude-based fusion to obtain multiple enhanced bit planes.

[0025] Multiple enhanced bit-plane level re-integrations yield an enhanced low-resolution image.

[0026] Furthermore, the specific method for multi-image enhanced bit-plane quasi-reintegration is as follows:

[0027] Multiple enhanced bit planes of each single channel are fused into an enhanced channel image for each channel;

[0028] The enhanced low-resolution image is obtained by fusing the enhanced channels of the red, green, and blue channels.

[0029] The beneficial effects of adopting the above technical solution are as follows: The spatial and frequency domain data augmentation method for image super-resolution provided by this invention, while preserving the spatial correlation of images, greatly enriches texture details by mining bit-level hidden information in low-resolution images and deeply fusing it with the frequency and amplitude features of high-resolution images. Compared with traditional methods, this invention can more effectively expand the value of training sets, with particularly significant improvement effects on large-structure network models, without destroying the physical structure of the image itself. Attached Figure Description

[0030] Figure 1 Flowchart of the spatial and frequency domain data enhancement method for image super-resolution provided in Embodiment 1 of the present invention;

[0031] Figure 2The bit plane fused from the full-band bit planes provided in Embodiment 1 of this invention comprises, (a) the first bit plane after full-band bit plane fusion, (b) the second bit plane after full-band bit plane fusion, (c) the third bit plane after full-band bit plane fusion, (d) the fourth bit plane after full-band bit plane fusion, (e) the fifth bit plane after full-band bit plane fusion, (f) the sixth bit plane after full-band bit plane fusion, (g) the seventh bit plane after full-band bit plane fusion, and (h) the eighth bit plane after full-band bit plane fusion.

[0032] Figure 3 The PSNR verification results after every 1000 training steps provided in Embodiment 1 of the present invention;

[0033] Figure 4 The SSIM verification results provided in Embodiment 1 of this invention after every 1000 training steps. Detailed Implementation

[0034] The specific embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. The following examples are only used to illustrate the technical concept of the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0035] Example 1:

[0036] A spatial and frequency domain data augmentation method for image super-resolution, such as Figure 1 As shown, it includes the following steps:

[0037] Step 1: Perform color channel stripping on high-resolution and low-resolution images of the same shooting scene, and parse them into multiple bit planes. Use Fourier transform to extract the frequency domain information of the high-resolution image, including frequency information and amplitude information.

[0038] Step 1.1: Decouple the color channels of the high-resolution image and the low-resolution image, and separate the single-channel images of the three primary colors of red, green and blue, respectively, to obtain the single-channel images of the high-resolution image and the single-channel images of the low-resolution image.

[0039] A single-channel image of a high-resolution image includes the red channel image, the green channel image, and the blue channel image of the high-resolution image; a single-channel image of a low-resolution image includes the red channel image, the green channel image, and the blue channel image of the low-resolution image.

[0040] Step 1.2: Perform planar decomposition on the single-channel image. Decompose each single-channel image into 8 independent bit planes according to the 8-bit depth to obtain the 8 bit planes of each single-channel image, namely the 1st bit plane to the 8th bit plane. Among them, the 1st bit plane to the 4th bit plane represent the high-frequency information of the single-channel image, and the 5th bit plane to the 8th bit plane represent the low-frequency information of the single-channel image.

[0041] Perform a Fast Fourier Transform (FFT) on all bit planes of the high-resolution image to extract frequency domain information that characterizes the image details, including frequency and amplitude information.

[0042] Step 2: Perform bit-plane transformation on the low-resolution image to obtain multiple reconstructed bit-planes;

[0043] For each low-resolution image, a permutation logic for the same weight bit plane between different channels is defined, including pairwise permutations and cyclic permutations between three channels, providing a total of 5 permutation logics. Pairwise permutations include red-green channel permutation, red-blue channel permutation, and blue-green channel permutation. Cyclic permutations between three channels include red, green, blue, red cyclic permutation and blue, green, red cyclic permutation.

[0044] In each channel, any number of bit planes are randomly selected from the 8 bit planes; based on 8-bit binary combinational logic, there are a total of 255 possible bit plane selections.

[0045] Based on the set swapping logic, a precise physical position swap is performed on the selected bit plane. After the swapping is completed, 24 reconstructed bit planes are obtained, including 8 reconstructed bit planes for each of the red, green and blue channels.

[0046] Step 3: Perform bit-plane fusion on multiple bit planes of the high-resolution image and multiple reconstructed bit planes of the low-resolution image to obtain multiple enhanced bit planes;

[0047] For high-resolution images and reconstructed low-resolution images, several transformation channels can be freely selected from the three independent channels of red, green and blue, including 3 single-channel selection methods, 3 dual-channel selection methods and 1 full-channel selection method;

[0048] For high-resolution images and reconstructed low-resolution images, an arbitrary number of bit planes are randomly selected from the selected transform channels for frequency domain information fusion; based on 8-bit binary combinational logic, there are a total of 255 possibilities for bit plane selection.

[0049] This embodiment adopts a full-band bit-plane fusion mode, which fuses the frequency domain information of the high-resolution image bit plane into the reconstructed bit plane corresponding to the low-resolution image between the selected channels. The fusion methods include frequency-based fusion and frequency and amplitude-based fusion, resulting in 24 enhanced bit planes.

[0050] The specific method for re-integrating multiple enhanced bit-plane level sensors is as follows:

[0051] Eight enhanced bit planes from each single channel are fused into one enhanced channel image;

[0052] The enhanced low-resolution image is obtained by fusing the enhanced channels of the red, green, and blue channels.

[0053] Step 4: Input the enhanced low-resolution image into the super-resolution network to generate the corresponding reconstructed image;

[0054] To verify the effectiveness of the spatial and frequency domain data augmentation methods for image super-resolution, this embodiment conducts comparative experiments using multiple bit-plane transformation schemes and multiple bit-plane fusion schemes. Bit-plane transformations are performed on low-resolution images based on multiple transformation schemes, and bit-plane fusion is performed on the bit-plane and reconstructed bit-plane of high-resolution images based on multiple fusion schemes, resulting in enhanced low-resolution images under various bit-plane transformation and fusion strategies. All possible bit-plane transformation and fusion strategies are exhaustively enumerated, and reconstructed images corresponding to various enhanced low-resolution images are generated using a super-resolution network. The quality of the reconstructed images is evaluated based on the high-resolution images, with Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) selected as performance evaluation metrics.

[0055] S1 constructs a first dataset based on several pairs of high-resolution and low-resolution images obtained, and selects one high-resolution image and one low-resolution image from the first dataset as the original image.

[0056] In this example, the RealSR dataset is used as the first dataset. The Canon_088 sample is selected from it. The color space of the high-resolution image Canon_088_HR and the low-resolution image Canon_088_LR4 of the Canon_088 sample is decoupled, and the single-channel images of the three primary colors of red (R), green (G), and blue (B) are extracted respectively, resulting in a total of 6 single-channel images, including 3 single-channel images of the high-resolution image Canon_088_HR and 3 single-channel images of the low-resolution image Canon_088_LR4. For the six single-channel images, a bit-plane decomposition technique was applied, decomposing each single-channel image into eight independent bit planes with an 8-bit depth. After the decomposition, 48 bit planes of the Canon_088 sample were obtained, including 24 bit planes of the high-resolution image Canon_088_HR and 24 bit planes of the low-resolution image Canon_088_LR4. Fast Fourier Transform (FFT) was performed on the 24 bit planes of the high-resolution image Canon_088_HR to extract frequency and amplitude information that characterizes image details.

[0057] This embodiment performs dimensionality transformation on each of the 24 bit planes of the high-resolution image Canon_088_HR and the low-resolution image Canon_088_LR4 extracted in step 1 to obtain a reconstructed low-resolution image, including 24 reconstructed bit planes. Full-band bit plane fusion is then performed on the reconstructed low-resolution image to obtain 24 enhanced bit planes. Finally, multiple enhanced bit planes are restored to an enhanced low-resolution image through level reintegration, such as... Figure 2 As shown, (a) is the first bit plane after full-band bit plane fusion, (b) is the second bit plane after full-band bit plane fusion, (c) is the third bit plane after full-band bit plane fusion, (d) is the fourth bit plane after full-band bit plane fusion, (e) is the fifth bit plane after full-band bit plane fusion, (f) is the sixth bit plane after full-band bit plane fusion, (g) is the seventh bit plane after full-band bit plane fusion, and (h) is the eighth bit plane after full-band bit plane fusion.

[0058] S2 establishes various bit-plane transformation strategies and conducts comparative experiments on bit-plane transformation strategies based on the original image;

[0059] The various bit-plane transformation strategies established in this example include a first transformation mode and a second transformation mode. The first transformation mode is a transformation mode within a single image channel, and the second transformation mode is a transformation mode between single image channels as described in step 2.

[0060] The first transformation mode includes:

[0061] For each low-resolution image, several channels are selected from the three independent channels of red, green, and blue. The methods for selecting channels include selecting a single channel, selecting a combination of two channels, and selecting all channels. Selecting a single channel means selecting one channel from the three channels of red, green, and blue, which includes three selection methods; selecting a combination of two channels means selecting two channels from the three channels of red, green, and blue, which includes three selection methods; selecting all channels means selecting the three channels of red, green, and blue, which includes one selection method, for a total of seven selection methods.

[0062] For the lower 5 bit planes representing high-frequency information in the selected channels (i.e., bit planes 1 to 5), their original arrangement order is broken and randomized to obtain 24 bit planes for low-resolution image reconstruction. In the first transformation mode, there are a total of 120 possible permutations of bit planes within each channel. Based on the channel selection and bit order reorganization in the first transformation mode, there are 840 possible transformations of the bit planes. After processing with the first transformation mode, the low-resolution image Canon_088_LR4 still maintains an output scale of 24 bit planes. All bit plane transformation results obtained from the first transformation mode are then reassembled into reconstructed low-resolution images to construct the first augmentation dataset. This includes 840 reconstructed low-resolution images;

[0063] The second transformation mode is the single-image channel transformation mode described in step 2 of this embodiment. In the second transformation mode, there are 1275 bit-plane transformation methods. All bit-plane transformation results obtained from the second transformation mode are reconstructed into a low-resolution image to build the second augmented dataset. This includes 1275 reconstructed low-resolution images;

[0064] This embodiment uses the HESR super-resolution network structure for experiments. The single-image inter-channel bit swap described in step 2 is selected, referred to as inter-channel swap 3all. All 5 possible channels are selected for swapping, and the 3rd bit plane is selected for swapping. That is, a total of 5 possible bit plane transformations are added to the HESR network and trained together with the original image. The experimental results are shown in Table 1. In the table, PSNR is the peak signal-to-noise ratio, and SSIM is the structural similarity. The larger the PSNR and SSIM values, the better the super-resolution restoration effect.

[0065] Table 1. Experimental results;

[0066]

[0067] S3 establishes multiple bit-plane fusion strategies and conducts comparative experiments on bit-plane fusion strategies based on multiple bit-planes of high-resolution images and multiple reconstructed bit-planes of low-resolution images obtained based on the second transform mode.

[0068] The various bit-plane fusion strategies established in this example include the first fusion mode, the second fusion mode, the third fusion mode, and the fourth fusion mode;

[0069] The first fusion mode is the full-band bit-plane fusion mode described in step 3 of this embodiment. In the first fusion mode, there are 3570 bit-plane fusion methods. All bit-plane fusion results obtained from the first fusion mode are reassembled into enhanced low-resolution images to construct a third enhanced dataset. This includes 3,570 enhanced low-resolution images;

[0070] The second mode is low-frequency structure fusion, and the specific method is as follows:

[0071] For high-resolution images and reconstructed low-resolution images, several transformation channels can be freely selected from the three independent channels of red, green and blue, including 3 single-channel selection methods, 3 dual-channel selection methods and 1 full-channel selection method;

[0072] For high-resolution images and reconstructed low-resolution images, an arbitrary number of bit planes are randomly selected from the 5th to 8th bit planes in the selected transform channels for frequency domain information fusion. Based on 8-bit binary combinational logic, there are a total of 16 possibilities for bit plane selection.

[0073] In the first fusion mode, the frequency domain information of the selected bit planes in the high-resolution image is fused to the selected reconstructed bit planes in the low-resolution image. The fusion methods for each selected bit plane include frequency-based fusion and frequency-and-amplitude-based fusion, resulting in 24 enhanced bit planes. In the second fusion mode, there are 1792 possible bit plane fusion methods. All the bit plane fusion results obtained in the second fusion mode are then reassembled into enhanced low-resolution images to construct the fourth enhancement dataset. This includes 1,792 enhanced low-resolution images;

[0074] The third fusion mode is high-frequency detail fusion, and the specific method is as follows:

[0075] For high-resolution images and reconstructed low-resolution images, several transformation channels can be freely selected from the three independent channels of red, green and blue, including 3 single-channel selection methods, 3 dual-channel selection methods and 1 full-channel selection method;

[0076] For high-resolution images and reconstructed low-resolution images, an arbitrary number of bit planes are randomly selected from the 1st to 4th bit planes in the selected transform channels for frequency domain information fusion. Based on 8-bit binary combinational logic, there are a total of 16 possibilities for bit plane selection.

[0077] In the first fusion mode, the frequency domain information of the selected bit planes in the high-resolution image is fused to the selected reconstructed bit planes in the low-resolution image. The fusion methods for each selected bit plane include frequency-based fusion and frequency-and-amplitude-based fusion, resulting in 24 enhanced bit planes. In the second fusion mode, there are 1792 possible bit plane fusion methods. All the bit plane fusion results obtained from the third fusion mode are then reassembled into enhanced low-resolution images to construct the fifth enhancement dataset. This includes 1,792 enhanced low-resolution images;

[0078] The fourth fusion mode combines low-frequency structure fusion with high-frequency detail fusion. The specific method is as follows:

[0079] For high-resolution images and reconstructed low-resolution images, several transformation channels can be freely selected from the three independent channels of red, green and blue, including 3 single-channel selection methods, 3 dual-channel selection methods and 1 full-channel selection method;

[0080] For high-resolution images and reconstructed low-resolution images, in the selected transform channels, any number of bit planes are randomly selected from the 1st to 4th bit planes and the 5th to 8th bit planes respectively for frequency domain information fusion.

[0081] In the first fusion mode, the frequency domain information of the selected bit planes in the high-resolution image is fused to the selected reconstructed bit planes in the low-resolution image. The fusion methods for each selected bit plane include frequency-based fusion and frequency-and-amplitude-based fusion, resulting in 24 enhanced bit planes. In the second fusion mode, there are 3584 possible bit plane fusion methods. All the bit plane fusion results obtained from the fourth fusion mode are then reassembled into enhanced low-resolution images to construct the sixth enhancement dataset. This includes 3,584 enhanced low-resolution images;

[0082] S4 inputs the four augmentation datasets into the super-resolution network, selects peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as performance evaluation metrics, and calculates the difference in peak signal-to-noise ratio and the difference in structural similarity between the images in the augmentation dataset and the original images, respectively.

[0083] The peak signal-to-noise ratio (PSNR) difference between the enhanced low-resolution image and the original image is calculated to obtain the degree of change of the original image by bit-plane fusion. The larger the PSNR difference, the greater the degree of change. The corresponding PSNR scale set is obtained. , , , ;

[0084] The structural similarity difference between the enhanced low-resolution image and the original image is calculated. This structural similarity difference reflects the degree to which bit-plane fusion alters the brightness, contrast, and structure of the original image, resulting in a corresponding set of structural similarity scales. , , , ;

[0085] This step aims to reorder the performance of all bit-plane transformation strategies using objective metrics, and then perform a uniform screening experiment on each scale set.

[0086] S5 obtains the maximum value of the peak signal-to-noise ratio difference from each peak signal-to-noise ratio scale set. Median and minimum value The corresponding fusion mode is used as a candidate bit-plane transformation scheme; the maximum value of the structural similarity difference is obtained from each set of structural similarity scales. Median and minimum value The corresponding fusion modes are used as candidate bit-plane fusion schemes; the maximum, median, and minimum values ​​represent the bit-plane transformation modes that have the greatest, moderate, and smallest degree of change to the original image, respectively.

[0087] S6 integrates the three representative candidate bit-plane transformation schemes selected in S5 into the super-resolution network for joint training; and combines the first dataset and the first augmented dataset. Second Enhanced Dataset The super-resolution network is input together; the new PSNR and SSIM values ​​of the output images after training are compared and arranged in descending order of value. , , , , , The recovery effect is characterized from best to worst. Based on the correspondence between recovery performance and bit transformation mode, the actual contribution of different bit plane transformation schemes to the super-resolution network model is determined and distinguished, thereby determining the optimal bit plane transformation scheme.

[0088] If the maximum value of the difference between PSNR and SSIM is P 01 S 01 If the corresponding bit-plane transformation scheme obtains performance peaks P1 and S1 after training, then P is determined. 01 S 01 The corresponding bit plane transformation scheme is the optimal bit plane transformation scheme under this scale.

[0089] If the minimum value of the difference between PSNR and SSIM is P03 S 03 If the corresponding bit-plane transformation scheme obtains performance peaks P1 and S1 after training, then P is determined. 03 S 03 The corresponding bit plane transformation scheme is the optimal bit transformation method under this scale.

[0090] If the median of the difference between PSNR and SSIM is P 02 S 02 If the corresponding bit-plane transformation scheme obtains performance peaks P1 and S1 after training, then the iterative optimization process is initiated: P... 02 S 02 Set the new search limit P' 01 S' 01 Compare P 01 With P 03 S 01 With S 03 The suboptimal results after training are used to set the solutions corresponding to P2 and S2 as the new lower bound P'. 03 S' 03 ; within the scale interval [P' 03 ,P' 01 ]、[S' 03 , S' 01 Re-extract the median point P' within the range. 02 S' 02 The verification process is iteratively executed to precisely identify the bit-plane transformation method with the highest recovery efficiency among the scale parameters. To improve the robustness of the model, not only is this optimal solution retained, but all candidate bit-plane transformation combinations with an error of 0.3 between the training result and the optimal PSNR value, and an SSIM value error within the range of 0.03, are also selected. This cluster of optimal solutions is then uniformly incorporated into the training process of the super-resolution network to achieve deep super-resolution enhancement of image details and textures.

[0091] Based on step 2, the full-band bit fusion mode described in step 3 is selected. Frequency domain information fusion is performed on the selected bit plane. The fusion channel is selected from all 7 possibilities. Full-band fusion is selected. The fusion bit plane is selected from 3 possibilities: the 1st bit plane, the 7th bit plane, or the 5th and 7th bit planes. The fusion method is frequency, abbreviated as HR_LR inter-channel fusion 57all. A total of 15 bit plane fusion results are obtained. The enhanced low-resolution image is then used for training in the super-resolution network. The experimental results are shown in Table 2. The larger the PSNR and SSIM values, the better the super-resolution restoration effect.

[0092] Table 2. Experimental results;

[0093]

[0094] Experiments show that the method provided in this embodiment can effectively expand the value of a limited dataset and improve the accuracy of reconstructed images, with PSNR and SSIM metrics significantly better than the original model.

[0095] This embodiment breaks through the limitations of traditional data augmentation, which only focuses on pixels or feature dimensions, and delves into the underlying bit level of the image. The core scheme includes three complementary augmentation strategies: random bit rearrangement within a single channel, bit swapping between channels, and frequency domain information fusion of the bit plane between high- and low-resolution images. By performing channel stripping and bit plane decomposition on both high-resolution and low-resolution images, the images are transformed into bit-level images. This refined operation effectively uncovers hidden texture details, establishes a correlation between underlying bit information and high-dimensional frequency domain features, and strengthens feature coupling between high- and low-resolution images without destroying the inherent spatial correlation of the images.

[0096] This implementation presents a spatial and frequency domain data augmentation method for image super-resolution, which offers significant technical advantages and application effects compared to traditional techniques. This method breaks through the limitations of traditional data augmentation, which only operates at the pixel or feature level, by introducing a cross-channel bit-plane transformation mode. It delves into the underlying bit level of the image, achieving cross-channel bit swapping. This refined bit-level operation effectively uncovers hidden bit-level texture and detail information in the image, greatly enriching the image's texture details without destroying its physical structure and inherent spatial correlations.

[0097] During the fusion phase, a full-band bit-plane fusion mode is used to deeply fuse the frequency and amplitude features of the high-resolution image into the bit-plane of the reconstructed low-resolution image. This frequency-domain fusion strategy establishes a strong correlation between the low-resolution low-level features and the high-resolution frequency-domain information, strengthens the feature coupling between high- and low-resolution images, and thus solves the problem of insufficient generalization ability caused by the disconnect between the degradation mode of the synthetic dataset and the real environment.

[0098] Experimental data show that this scheme can effectively expand the value of a limited training set, and its improvement is particularly significant in larger network models. Figure 3 As shown, after adopting the HR_LR inter-channel fusion 57all strategy, the peak signal-to-noise ratio (PSNR) of the enhanced low-resolution image increased from 34.096 to 34.453; Figure 4 As shown, the structural similarity SSIM improved from 0.9099 to 0.9140, and all indicators were significantly better than the original model, achieving deep super-resolution enhancement of image details and textures.

[0099] Example 2:

[0100] This embodiment proposes an electronic device, including: one or more processors, and a memory for storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform the spatial and frequency domain data enhancement method for image super-resolution.

[0101] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program that, when executed by the processor, implements the spatial and frequency domain data enhancement method for image super-resolution as described in the embodiments. It is understood that the electronic device may also include an input / output (I / O) interface and communication components.

[0102] The processor is used to execute all or part of the steps in the spatial and frequency domain data enhancement method for image super-resolution as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0103] The processor may be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the spatial and frequency domain data enhancement method for image super-resolution described in the above embodiments.

[0104] Example 3:

[0105] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0106] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the spatial and frequency domain data enhancement methods for image super-resolution described in the various embodiments of this application.

[0107] The aforementioned storage media include: flash memory, hard disks, multimedia cards, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disks, optical discs, servers, APP (Application) application stores, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the spatial and frequency domain data enhancement methods for image super-resolution described above.

[0108] Example 4:

[0109] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned spatial and frequency domain data enhancement method for image super-resolution.

[0110] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0111] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0112] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this disclosure and its equivalents, then the intent of this disclosure also includes these modifications and variations.

Claims

1. A spatial and frequency domain data enhancement method for image super-resolution, characterized in that, Includes the following steps: High-resolution and low-resolution images of the same shooting scene are analyzed into multiple bit planes; frequency domain information of the high-resolution image is extracted. Perform bit-plane transformation on the low-resolution image to obtain multiple reconstructed bit-planes; Bit-plane fusion is performed on multiple bit planes of a high-resolution image and multiple reconstructed bit planes of a low-resolution image to obtain an enhanced low-resolution image with fused frequency domain information; The low-resolution image is enhanced and input into the super-resolution network to generate the corresponding reconstructed image.

2. The spatial and frequency domain data enhancement method for image super-resolution according to claim 1, characterized in that, The specific method for parsing high-resolution and low-resolution images of the same shooting scene into multiple bit planes is as follows: The color channels of the high-resolution and low-resolution images are decoupled separately, and each high-resolution and low-resolution image is decomposed into three single-channel images: red, green and blue. Each single-channel image is decomposed into planes, which are divided into 8 bit planes, namely the 1st bit plane to the 8th bit plane. The 1st bit plane to the 4th bit plane represent the high-frequency information of the single-channel image, and the 5th bit plane to the 8th bit plane represent the low-frequency information of the single-channel image.

3. The spatial and frequency domain data enhancement method for image super-resolution according to claim 1, characterized in that, The specific method for extracting frequency domain information from high-resolution images is as follows: A fast Fourier transform is performed on all bit planes of the high-resolution image to extract the frequency domain information of the high-resolution image, which includes frequency information and amplitude information.

4. The spatial and frequency domain data enhancement method for image super-resolution according to claim 1, characterized in that, The specific method for performing bit-plane transformation on low-resolution images is as follows: For each low-resolution image, set the logic for swapping bit planes with the same weight between different channels; In each channel, any number of bit planes are randomly selected from the 8 bit planes. Based on the defined swapping logic, precise physical position swaps are performed on the selected bit plane to obtain multiple reconstructed bit planes.

5. The spatial and frequency domain data enhancement method for image super-resolution according to claim 4, characterized in that, The interchange logic includes pairwise interchanges and cyclical permutations between the three channels; pairwise interchanges include red-green channel interchange, red-blue channel interchange, and blue-green channel interchange; cyclical permutations between the three channels include red, green, blue, red in sequence and blue, green, red, blue in sequence.

6. The spatial and frequency domain data enhancement method for image super-resolution according to claim 1, characterized in that, The specific method for bit-plane fusion is as follows: For reconstructing a low-resolution image from a high-resolution image, several channels can be freely selected from the three independent channels of red, green, and blue. For high-resolution images and reconstructed low-resolution images, an arbitrary number of bit planes are randomly selected from the selected channels for frequency information fusion. Between each selected channel, the frequency domain information of the high-resolution image bit plane is fused to the reconstructed bit plane corresponding to the low-resolution image. The fusion methods include frequency-based fusion and frequency and amplitude-based fusion to obtain multiple enhanced bit planes. Multiple enhanced bit-plane level re-integrations yield an enhanced low-resolution image.

7. A spatial and frequency domain data enhancement method for image super-resolution according to claim 6, characterized in that, The specific method for re-integrating multiple enhanced bit-plane level sensors is as follows: Multiple enhanced bit planes of each single channel are fused into an enhanced channel image for each channel; The enhanced low-resolution image is obtained by fusing the enhanced channels of the red, green, and blue channels.

Images