A resampling method for super-high-definition image enhancement guided by model perception

Abstract

The application relates to the technical field of image enhancement, and discloses a resampling method for guiding super-high-definition image enhancement through model perception, which comprises the following steps: inputting a super-high-definition image into a trained resampling model to obtain a resampled image; the training process of the resampling model comprises the following steps: performing downsampling on the super-high-definition image to obtain a low-resolution image; inputting the super-high-definition image into an LMAR module to obtain a compensation embedded image; the LMAR module comprises compensation kernel estimation and steganographic resampling; performing convolution operation on the low-resolution image and the compensation embedded image after splicing to obtain the resampled image; and the loss function used for training the resampling model comprises a perception loss and an adversarial loss; the resampling method in the application significantly improves the performance under low resolution, meanwhile, equivalent results are maintained under the original resolution, and the enhancer does not need to be retrained.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, and more specifically to a resampling method for ultra-high-definition image enhancement guided by model perception. Background Technology

[0002] Over time, learning-based image enhancement algorithms have achieved significant performance improvements. However, most existing methods are incompatible with ultra-high-definition (UHD) images because the megapixel count of UHD images imposes a heavy computational burden. Ultra-high-definition (UHD) images generally refer to images with a resolution of 3840x2160 or higher. In this invention, low-resolution images refer to images with a resolution less than 3840x2160.

[0003] To address this issue, previous work has mitigated computational overhead by employing downsampling to reduce resolution. Specifically, ultra-high-resolution input images are typically downsampled to a smaller size for enhancement, and then upsampled back to the original resolution. Notably, most research has focused on designing the internal algorithms, with little attention paid to the initial and final resampling operators.

[0004] Re-examining the ultra-high-definition image enhancement process above—downsampling-enhancement-upsampling—a significant issue arises when the internal enhancement algorithm remains unchanged: how much do these two resampling operations affect the final enhancement result? On one hand, as the degree of downsampling increases, the quality of the final enhancement result tends to decrease; on the other hand, different interpolation functions produce a wide variety of enhancement results. This phenomenon can be attributed to a lack of correlation between the resampling operator and the enhancement algorithm. Therefore, a model-aware resampling method is needed to guide ultra-high-definition image enhancement. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a resampling method for ultra-high-definition image enhancement guided by model perception. This method can associate resampling operators with enhancement algorithms, activating the potential of resampling operations without changing the enhancement algorithms. This results in a significant improvement in the enhancement effect of low-resolution images while maintaining consistency in the enhancement effect of the original resolution input.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A resampling method for ultra-high-resolution image enhancement guided by model perception is proposed. The ultra-high-resolution image is input into a trained resampling model to obtain a resampling image. The training process of the resampling model includes the following steps:

[0008] Step 1: Downsample the ultra-high-resolution image x to obtain the low-resolution image x. d ;

[0009] Step two involves inputting the ultra-high-resolution image into the LMAR module to obtain a compensated embedded image. The LMAR module includes compensation kernel estimation and steganalytic resampling. The compensation kernel estimation is used to customize a compensation convolution kernel K for different inputs and resampling scales through implicit neural representations. The steganalytic resampling performs a convolution operation between the compensation convolution kernel and the ultra-high-resolution image to obtain compensation information, which is then embedded into a low-resolution image to obtain the compensated embedded image x. c ;

[0010] Step 3: After concatenating the low-resolution image and the compensated embedded image, perform a convolution operation to obtain the resampled image;

[0011] Step four, the loss function used to train the resampling model includes perceptual loss L. c and combat losses L d ; Counteracting losses L d Used to constrain low-resolution images x d and compensation embedded image x c Distribution consistency; perceptual loss L c Including resampling consistency loss L rc Feature attention loss L fa and generation loss L g .

[0012] Furthermore, the compensation kernel estimation specifically includes the following steps:

[0013] S11 uses scale-dependent features to estimate the compensation kernel; two scale-dependent features are employed, including: relative coordinate grid and pixel unit scale.

[0014] The relative coordinate grid describes the pixel position offset during resampling; first, the normalized coordinates C of the downsampling scale are calculated. d and the normalized coordinates C at ultra-high definition scale u :

[0015]

[0016]

[0017] In the formula, i∈[0, h-1]V[0, H-1], j∈[0, w-1]V[0, W-1] represent the position indices in the length and width directions of the image, respectively; h and w represent the height and width of the low-resolution image, respectively, and H and W represent the height and width of the ultra-high-resolution image, respectively.

[0018] S12, due to the normalized coordinates C of the downsampling scale d and the normalized coordinates C at ultra-high definition scale u Since the coordinates are inconsistent, the relative coordinate grid offset is calculated through projection. The specific calculation formula is as follows:

[0019]

[0020]

[0021]

[0022] Where gridsample represents the remapping function, which maps C... d Projected onto a coordinate system of size H×W Represents relative coordinate offset, C r It is the relative coordinate offset under the downsampling scale;

[0023] S13, pixel unit ratio P c (i, j) represents the pixel area change during upsampling and downsampling:

[0024]

[0025] All pixel locations share the same pixel unit ratio;

[0026] S14, using backprojection as a constraint to capture the relationship between each low-resolution input and its corresponding ultra-high-resolution image:

[0027]

[0028] Where Conv represents the convolutional layer; U(·) represents the upsampling operator, which is used to upsample a low-resolution image to a full-resolution image with the same resolution as the ultra-high-definition image; The back projection represents the ultra-high definition image, where H and W represent the height and width of the ultra-high definition image;

[0029] S15, by offsetting the relative coordinates C r Pixel unit ratio P c and back projection The input is fed into a multilayer perceptron to predict the compensated convolutional kernel for each pixel location:

[0030]

[0031] Where MLP stands for Multilayer Perceptron, [;] represents feature concatenation along the channel dimension, and K represents the compensation convolution kernel at each pixel location.

[0032] Furthermore, the steganalytic resampling includes the following steps:

[0033] S21, after obtaining the compensation convolution kernels for different images, perform a convolution operation between the compensation convolution kernel K and the ultra-high-definition image x to calculate the compensation information:

[0034]

[0035] in, Representative compensation information, Represents the convolution operation;

[0036] S22, In order to further preserve the intrinsic features of the low-resolution image, the compensation information is downsampled and compensated steganography is performed in the low-resolution space:

[0037]

[0038] Where, x c To compensate for the embedded image, Conv represents a 1×1 convolutional layer; D(·) represents a downsampling operator used to downsample the ultra-high resolution compensation information to low resolution compensation information; and [;] represents feature concatenation along the channel dimension.

[0039] Furthermore, the adversarial loss L d =log(D(x) d ))+log(1-D(x c )), D(·) represents the discriminator.

[0040] Furthermore, perceived loss L c =L rc +αL fa +βL g ;

[0041] Among them, the sampling consistency loss L rc For: L rc =SmoothL1(E(x)-U(E(x)) c SmoothL1(·) represents the Smooth L1 loss function, and E(·) represents the bottleneck feature.

[0042] Feature attention loss L fa for:

[0043] Generation loss L g =log(D(x) c D(·) represents the discriminator;

[0044] α and β are adjustable weight parameters.

[0045] Compared with the prior art, the beneficial technical effects of the present invention are:

[0046] This invention addresses the issue that current ultra-high-definition image enhancement methods employ a downsampling-enhancement-upsampling paradigm, resulting in poor enhancement performance for low-resolution images. It proposes a resampling method for ultra-high-definition image enhancement guided by model perception. This framework mainly consists of two parts: compensation kernel estimation and steganalytic resampling. Compensation kernel estimation aims to customize convolutional kernels for different inputs and resampling scales through implicit neural representations. Steganalytic resampling utilizes the obtained convolutional kernels to perform convolution operations with the input image, obtaining compensation information, which is then embedded into the low-resolution image, ensuring consistent feature representation between the low-resolution image and the full-resolution input. Experimental results show that the resampling method in this invention significantly improves performance at low resolution while maintaining equivalent results at the original resolution without retraining the enhancer. Attached Figure Description

[0047] Figure 1 This is a schematic diagram illustrating the innovation of the resampling method of the present invention;

[0048] Figure 2 This is a schematic diagram illustrating the effects of downsampling and upsampling as described in this invention;

[0049] Figure 3 A schematic diagram illustrating the method design of this invention;

[0050] Figure 4 This is a schematic diagram of the resampling model and image enhancement model used in this invention. Detailed Implementation

[0051] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0052] This invention first delves into the impact of resampling from the perspective of the model, where resampling directly disrupts the model's ability to represent intermediate features, such as... Figure 2 As shown. Therefore, the intermediate features of ultra-high-resolution images provide valuable insights into how to perform resampling in a model-aware manner.

[0053] like Figure 3 and Figure 4 As shown, this invention proposes a model framework comprising two core ideas: compensation kernel estimation and steganalytic resampling. Compensation kernel estimation aims to customize convolutional kernels for different inputs and different resampling scales through implicit neural representations. Steganalytic resampling aims to use the obtained convolutional kernels to perform convolution operations with the input image to obtain compensation information, which is then embedded into the low-resolution image, ensuring consistent feature representations between the low-resolution and ultra-high-resolution images.

[0054] This embodiment presents a model-aware-guided resampling method for ultra-high-definition image enhancement, aiming to associate resampling operators with enhancement algorithms. Unlike the downsampling-enhancement-upsampling paradigm, this invention, without altering the enhancement algorithm, combines resampling operators and enhancement algorithms to significantly improve the enhancement effect of low-resolution images while maintaining consistency in the enhancement effect of the original resolution input. The specific implementation steps are as follows:

[0055] Step 1: Compensation Assessment

[0056] Step 1.1: The image resampling process based on interpolation can be decomposed into projection grid calculation and pixel weighted aggregation. Therefore, this invention uses scale-related features to estimate the compensation kernel. This invention employs two scale features: relative coordinate grid and pixel unit scale. The relative coordinate grid describes the pixel position offset in the resampling process. First, the normalized coordinates at the downsampling scale and the ultra-high resolution scale are calculated, denoted as C0. d ∈R h×w×2 (h, w represent the height and width of the low-resolution image) and C u ∈R H×W×2 (H and W represent the height and width of the ultra-high-definition image), and the calculation process is specifically represented as follows:

[0057]

[0058]

[0059] In the formula, i∈[0, h-1]V[0, H-1] and j∈[0, w-1]V[0, W-1] represent the position indices in the length and width directions of the image, respectively.

[0060] Step 1.2: Since the coordinates of the two scales obtained from the calculation are inconsistent, the relative coordinate grid offset is calculated by projection. The specific calculation formula is as follows:

[0061]

[0062]

[0063]

[0064] Where gridsample represents the remapping function, which maps C... d Projected onto a size of H×W Represents relative coordinate offset, C r It is the relative coordinate offset under the downsampling scale.

[0065] Step 1.3: The pixel unit ratio represents the change in pixel area during upsampling and downsampling. The specific calculation formula is as follows:

[0066]

[0067] Among them, P c (i, j) ∈ R H×W×2 All pixel locations share the same pixel unit ratio.

[0068] Step 1.4: The scale-related features mentioned above are the same for all samples at a given scale, resulting in the generation of the same compensation kernel without considering the differences between different images. Therefore, this invention proposes to use backprojection as a constraint to capture the relationship between each low-resolution input and its corresponding full-resolution image, specifically calculated as follows:

[0069]

[0070] Where Conv represents a 1×1 convolutional layer, x∈R H×W×3 Represents ultra-high definition input image, x d ∈R h×w×3 U represents a low-resolution image, and U(·) represents the upsampling operator, which upsamples the low-resolution image to an ultra-high-resolution image. Represents the back projection of an ultra-high-definition input image.

[0071] Step 1.5: By inputting the relative coordinate offset, pixel unit ratio, and backprojection features into the multilayer perceptron, the compensated convolutional kernel for each pixel position is predicted. The calculation formula is as follows:

[0072]

[0073] Where MLP represents a multilayer perceptron, [;] represents feature concatenation along the channel dimension, and K∈R HW×3×3×9 The compensation convolution kernel represents the position of each pixel.

[0074] Step 2, Stegographic Resampling

[0075] Step 2.1: After obtaining the specific compensation kernels for different images, convolve them with the input ultra-high-definition image to calculate the compensation information, as shown below:

[0076]

[0077] in, Represents ultra-high definition compensation information. This represents the convolution operation.

[0078] Step 2.2: To further preserve the intrinsic features of the low-resolution image, this invention downsamples the ultra-high-definition compensation information and performs compensatory steganography in the low-resolution space. The specific formula is as follows:

[0079]

[0080] Where, x c For learnable low-resolution images, also known as compensated embedded images, Conv represents a 1×1 convolutional layer, D(·) represents a downsampling operator that downsamples the ultra-high resolution compensation information into low-resolution compensation information, and [;] represents feature concatenation along the channel dimension.

[0081] Step 3: The model optimization objective includes resampling consistency loss, which is expressed as:

[0082] L rc =SmoothL1(E(x)-U(E(x)) c )));

[0083] Where E(x) represents the bottleneck feature of the ultra-high-definition image, E(x) c ) represents the bottleneck features of a learnable low-resolution image.

[0084] The optimization objective also includes feature attention loss, expressed as:

[0085]

[0086] Here, ||·||² represents L2 regularization, and the above loss aims to embed compensation information of model preferences into the learnable low-resolution image x. c In the middle, but ignoring the original low-resolution image x d and learnable low-resolution image x c Consistency in distribution among them.

[0087] Therefore, this invention introduces an adversarial loss, calculated as follows:

[0088] L d =log(D(x) d ))+log(1-D(x c ));

[0089] Where D(·) is the discriminator, which ensures the interaction between the resampling operator and the model while preserving x. c and x d Consistency in distribution among them.

[0090] Simultaneously, due to the introduction of the discriminator, a new generation loss term is added, which is expressed as:

[0091] L g =log(D(x) c ));

[0092] In summary, the overall optimization objective is divided into two parts. The first optimization objective focuses on guiding resampling through model awareness, and is expressed as:

[0093] L c =L rc +αL fa +βL g ;

[0094] Here, α and β are adjustable parameters used to balance different losses, and the second optimization objective is to overcome the adversarial loss L. d To constrain distribution consistency.

[0095] Step 4: Model Pre-training. This invention first trains multiple benchmark models on ultra-high-definition datasets to demonstrate the scalability of its method. For all benchmark models, this invention adopts a common encoder-decoder architecture, including two downsampling operations and two upsampling operations. Downsampling is implemented using convolutional layers with a stride of 2, while upsampling is implemented using transposed convolutional layers. This invention trains three benchmark models: the CNN-I model on the UHD-LOL4K dataset, the CNN-H model on the 4KIL dataset, and the Restormer model on the 4KIL dataset. For the CNN-I model, the INV module is used as the basic module of the convolutional neural network. For the CNN-H model, the HIN module is used as the basic module of the convolutional neural network. The Adam optimizer is used to optimize the overall network, and cosine annealing is used as the learning strategy. For both CNN models, the initial learning rate is 1e-4, the batch size is set to 4, and randomly cropped 1024×1024 image regions are selected as training pairs. The CNN-I model is trained for 100 epochs, and the CNN-H model is trained for 30 epochs. For Restormer, the batch size was set to 1, and randomly cropped 768×768 image regions were selected as training pairs. The initial learning rate was 5e-4, and the number of training rounds was 30. The entire experiment was conducted on a single NVIDIA GeForce RTX3090Ti graphics card.

[0096] Step 5: Model-aware guided resampling training. After obtaining the pre-trained baseline model, the resampling operator and the model are associated through retraining. The parameters of the pre-trained baseline model remain unchanged; only the newly added convolutional layers and multilayer perceptrons are trained. Considering the randomness of scale variation during resampling, this invention employs stochastic scale training. Specifically, this invention first crops an H-shaped image from the ultra-high-resolution image. P ×W p The image. Next, the present invention randomly samples and resamples from a uniform distribution using a scaling factor r. Finally, the present invention applies the scaling factor r to the high-resolution H image. P×W p The image is downsampled to generate a low-resolution h. P ×w p Image. The Adam optimizer is used to optimize the newly added learnable parameters, and cosine annealing is used as the learning strategy. For both CNN models, H... P ×W p The resolution was set to 1024×1024, the initial learning rate was set to 4e-4, the CNN-I model was trained for 12 epochs, and the CNN-H model was trained for 24 epochs. For the Restormer model, H... P ×W p The image size is set to 768×768, the initial learning rate is set to 4e-4, and the training epochs are 60. For the discriminator, the initial learning rate is 2e-4. In this training epoch, the batch size of all models is set to 1.

[0097] Figure 4 In the diagram, (a) represents the training process of the model. Figure 4 (b) in the text represents the reasoning process. Figure 4 (c) in the diagram represents a schematic of the LMAR module.

[0098] To quantitatively evaluate the effectiveness of this invention, it was combined with various resampling operators, and its robustness was tested at different scales. PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index Metric) were chosen as numerical evaluation metrics. In the quantitative analysis, this invention is adaptable to various benchmark models, and when combined with various resampling operators, it consistently improves performance across different resampling resolutions.

[0099] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0100] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A resampling method for ultra-high-definition image enhancement guided by model perception, wherein the ultra-high-definition image is input into a trained resampling model to obtain a resampling image; the training process of the resampling model includes the following steps: Step 1: Process the ultra-high-definition image Downsampling is performed to obtain a low-resolution image. ; Step two involves inputting the ultra-high-resolution image into the LMAR module to obtain the compensated embedded image. The LMAR module includes compensation kernel estimation and steganalytic resampling. The compensation kernel estimation is used to customize the compensation convolutional kernel for different inputs and resampling scales through implicit neural representations. Specifically, it includes: Scale-dependent features are used to estimate the compensation kernel; two scale-dependent features are employed, including: relative coordinate grid and pixel unit scale. The relative coordinate grid describes the pixel position offset in the resampling process. First, the normalized coordinates at the downsampling scale and the normalized coordinates at the ultra-high resolution scale are calculated. Since the normalized coordinates at the downsampling scale and the normalized coordinates at the ultra-high definition scale are not the same in size, the relative coordinate offset is calculated by projection. The pixel unit ratio represents the change in pixel area during upsampling and downsampling, and all pixel locations share the same pixel unit ratio; Back projection is used as a constraint to capture the relationship between each low-resolution input and its corresponding ultra-high-resolution image; By inputting relative coordinate offset, pixel unit ratio, and back projection into a multilayer perceptron, a compensated convolutional kernel for each pixel position is predicted. The steganalytic resampling involves convolving the ultra-high-resolution image with the compensated convolution kernel to obtain compensation information, which is then embedded into the low-resolution image to obtain the compensated embedded image. Specifically, it includes: After obtaining the compensation convolution kernels for different images, the compensation convolution kernels are convolved with the ultra-high-definition images to calculate the compensation information. To further preserve the intrinsic features of the low-resolution image, the compensation information is downsampled and compensated steganography is performed in the low-resolution space to obtain the compensated embedded image. Step 3: After concatenating the low-resolution image and the compensated embedded image, perform a convolution operation to obtain the resampled image; Step four, the loss function used to train the resampling model includes perceptual loss. and combat losses ; combating losses Used to constrain low-resolution images and compensation embedded image Distribution consistency; perceived loss Including resampling consistency loss Feature attention loss and generation loss .

2. The resampling method for ultra-high-definition image enhancement guided by model perception according to claim 1, characterized in that, The relative coordinate grid describes the pixel position offset during resampling. First, normalized coordinates at the downsampling scale and normalized coordinates at the ultra-high resolution scale are calculated, including: ； ； In the formula, , These represent the position indices in the length and width directions of the image, respectively. These represent the height and width of the low-resolution image, respectively. These represent the height and width of the ultra-high-definition image, respectively. Normalized coordinates for the downsampling scale. Normalized coordinates for ultra-high definition scale; Since the normalized coordinates at the downsampling scale and the normalized coordinates at the ultra-high definition scale are not of the same magnitude, the relative coordinate offset is calculated through projection, specifically including: ； ； ； in, This represents the remapping function, which will... Projected onto a size of coordinates , Represents relative coordinate offset. It is the relative coordinate offset under the downsampling scale; The pixel unit ratio represents the pixel area change during upsampling and downsampling, including: ； All pixel locations share the same pixel unit ratio. ; The method of using back projection as a constraint to capture the relationship between each low-resolution input and its corresponding ultra-high-resolution image includes: ； in, The representative convolutional layer; This represents an upsampling operator used to upsample a low-resolution image to a full-resolution image with the same resolution as an ultra-high-definition image; Reverse projection representing ultra-high definition images, Represents the height and width of an ultra-high-definition image; The process of predicting the compensated convolutional kernel for each pixel location by inputting relative coordinate offset, pixel unit ratio, and back projection into a multilayer perceptron includes: ； in, Represents a multilayer perceptron. Feature splicing representing the channel dimension The compensation convolution kernel represents the position of each pixel.

3. The resampling method for ultra-high-definition image enhancement guided by model perception according to claim 1, characterized in that, After obtaining the compensation convolution kernels for different images, the compensation convolution kernels are convolved with the ultra-high-definition images to calculate the compensation information, including: ； in, Representative compensation information, Represents the convolution operation; To compensate for the convolution kernel, For ultra-high definition images; To further preserve the intrinsic features of the low-resolution image, the compensation information is downsampled and compensated steganography is performed in the low-resolution space to obtain a compensated embedded image, including: ； in, To compensate for the embedded image, represent Convolutional layers; This represents a downsampling operator used to downsample ultra-high-definition compensation information to lower-resolution compensation information. Feature splicing representing the channel dimension.

4. The resampling method for ultra-high-definition image enhancement guided by model perception according to claim 1, characterized in that, The resistance loss , This indicates a discriminator.

5. The resampling method for ultra-high-definition image enhancement guided by model perception according to claim 1, characterized in that, Perceived loss ; Among them, sampling consistency loss for: ; Describes the Smooth L1 loss function. Indicates the characteristics of computational bottlenecks; Feature attention loss for: ; Generation loss , Indicates the discriminator; and It is an adjustable weight parameter.

Images