Image super-resolution sharpness processing method and device and storage medium

Abstract

The invention discloses an image super-resolution sharpness processing method and device, and a storage medium. The method comprises the following steps: S1, obtaining a to-be-processed low-resolution image and a target magnification; s2, performing interpolation processing on the low-resolution image to obtain a condition image; s3, constructing a conditional diffusion model and training the conditional diffusion model, wherein the conditional diffusion model comprises a forward noise adding network and a reverse denoising network; and S4, obtaining a random noise image, splicing the random noise image with the condition image, inputting the spliced image into a reverse denoising network in the trained condition diffusion model, carrying out iterative denoising, and outputting a high-resolution image. The problems that traditional interpolation and an existing deep network cannot adapt to non-integer multiplying power at the same time, robustness to real degradation is poor, and reconstruction details are missing are solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image super-resolution enhancement processing method, device, and storage medium. Background Technology

[0002] The core problem of image super-resolution is solving the problem of information loss. Low-resolution images with information loss are usually caused by limitations of image acquisition equipment (such as cameras and sensors), transmission bandwidth limitations, or storage compression. Image super-resolution technology aims to recover high-resolution details from low-resolution images. Currently, the mainstream methods are mainly divided into traditional methods and deep learning-based methods. Traditional methods include interpolation methods such as bilinear interpolation and bicubic interpolation, which improve resolution through pixel interpolation; and reconstruction-based methods such as sparse representation and regularization methods, which utilize prior knowledge of the image, such as sparsity and smoothness, for reconstruction. Deep learning-based methods typically use CNN-based super-resolution models like SRCNN, which learn the low-resolution to high-resolution mapping through simple convolutional networks, as well as VDSR, RCAN, etc., which improve performance and enhance detail recovery capabilities through deeper network structures, residual learning, or the introduction of channel attention mechanisms.

[0003] However, existing technologies often fail to recover high-frequency details using traditional interpolation methods, resulting in blurred image edges. These methods also have limited ability to handle complex textures and details and cannot adapt to complex degradations in real-world scenes, such as noise and blur. Reconstruction-based methods rely on manually designed prior knowledge, making them ill-suited for complex image content, sensitive to noise and compression artifacts, and exhibiting unstable recovery results. Deep learning-based methods perform poorly in scenarios outside of the training data and have limited adaptability to different types of degradation, such as noise, blur, and compression artifacts. Summary of the Invention

[0004] To address the above problems, this invention provides an image super-resolution sharpening method, the specific technical solution of which is as follows: S1: Obtain the low-resolution image to be processed and the target magnification; S2: Interpolate the low-resolution image to obtain the conditional image; S3: Construct and train a conditional diffusion model, which includes a forward noise-adding network and a backward noise-reducing network; S4: Obtain a random noise image and stitch it together with the conditional image. Input the stitched image into the inverse denoising network in the trained conditional diffusion model for iterative denoising and output a high-resolution image.

[0005] Furthermore, in step S2, the interpolation process yields the conditional image, as detailed below: The low-resolution image is interpolated to the same initial pixel size as the target high-resolution image to obtain a conditional image.

[0006] Furthermore, the reverse denoising network is a U-Net architecture, and an LSTM module is embedded in the jump connection path of the U-Net.

[0007] Used to pass hidden states between adjacent time steps to improve detail consistency.

[0008] Furthermore, in step S3, the training of the conditional diffusion model is specifically as follows: S301: Construct a high-resolution image training set and downsample the high-resolution images to obtain corresponding low-resolution images; S302: Gaussian noise is gradually added to the high-resolution image through the forward noise-adding network until a noisy image is obtained; S303: The low-resolution image obtained in step S301 is interpolated to the same size as the corresponding high-resolution image and used as a condition. It is then stitched together with the noisy image at the current time step and input into the inverse denoising network to train the inverse denoising network to predict the added noise.

[0009] Furthermore, in step S301, the high-resolution image is downsampled at at least two different downsampling ratios to form multi-ratio sample pairs to support super-resolution inference at the arbitrary real-number ratio.

[0010] Furthermore, in step S301, the downsampling factor includes at least two of the integer factors 4×, 8×, and 16×, and the non-integer factors 3×, 5×, and 7×, so that the model learns a factor-independent degenerate representation in the feature space.

[0011] Furthermore, in step S303, a weighted combination of L1 loss, L2 loss and negative SSIM loss is used as the loss function for iterative training; The loss function is expressed as follows: .

[0012] Furthermore, after step S4, step S5 is also included: restoring the pixel values ​​of the generated high-resolution image to the dynamic range of the original image.

[0013] The present invention also provides an image super-resolution enhancement processing device, the image super-resolution enhancement processing device comprising: a memory, a processor, and an image super-resolution enhancement processing program stored in the memory and executable on the processor, wherein the image super-resolution enhancement processing program, when executed by the processor, implements the steps of the image super-resolution enhancement processing method described above.

[0014] The present invention also provides a computer storage medium storing an image super-resolution enhancement processing program, wherein when the image super-resolution enhancement processing program is executed by a processor, it implements the steps of the image super-resolution enhancement processing method described above.

[0015] The beneficial effects of this invention are as follows: 1. This invention achieves high-precision super-resolution reconstruction of arbitrary real-value magnification using a single model by acquiring low-resolution images and target magnification ratios, interpolating to generate conditional images, constructing and training a conditional diffusion model containing forward noise addition and reverse denoising networks, and then concatenating random noise with the conditional images and inputting them into the reverse denoising network for iterative denoising. It can output high-definition images with sharp edges, natural textures, and no need for post-processing, solving the problems of traditional interpolation and existing deep networks being unable to adapt to non-integer magnification ratios, having poor robustness to real degradation, and missing reconstruction details.

[0016] 2. The conditional diffusion model of this invention includes a forward noise-adding network and a reverse noise-denoising network. By embedding the low-resolution image into the reverse noise-denoising path throughout the process, the structural drift problem caused by random generation of unconditional diffusion is solved, and the geometric consistency at high magnification is improved. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0018] Figure 2 This is a schematic diagram of the noise addition and denoising process in the conditional diffusion model.

[0019] Figure 3 This is a schematic diagram of the denoising network framework. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention are clearly and completely described in the following description. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] In the description of the embodiments of the present invention, it should be noted that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of the invention is conventionally placed during use, or the orientation or positional relationship in which those skilled in the art conventionally understand it during use. This is only for the convenience of describing the present invention and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of the present invention. Furthermore, the terms "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0023] Example 1 Embodiment 1 of the present invention discloses an image super-resolution enhancement processing method, such as... Figure 1 and Figure 2 As shown, the specific steps are as follows: S1: Obtain the low-resolution image to be processed and the target magnification; Specifically, the target magnification is any real number greater than 1.

[0024] S2: Interpolate the low-resolution image to obtain a conditional image; In a preferred embodiment, the low-resolution image is interpolated to the same initial pixel size as the target high-resolution image to obtain the conditional image.

[0025] S3: Construct and train a conditional diffusion model, which includes a forward noise-adding network and a backward noise-reducing network; The inverse denoising network takes the conditional image as a conditional input and is continuously guided by the conditional image during the iterative denoising process.

[0026] In a preferred embodiment, the reverse denoising network is a U-Net architecture, and an LSTM module is embedded in the jump connection path of the U-Net. Specifically, both the encoder and decoder of the U-Net include a normalization layer and a channel attention layer to accelerate convergence and enhance feature representation capabilities.

[0027] As a preferred embodiment, the training of the conditional diffusion model is specifically as follows: S301: Construct a high-resolution image training set and downsample the high-resolution images to obtain corresponding low-resolution images; In this embodiment, the high-resolution image training set is constructed as follows: High-resolution image datasets are obtained by using high-resolution cameras or public datasets such as the NWPU dataset. The corresponding low-resolution image is generated by downsampling the high-resolution image dataset. The downsampling factor is determined according to the required image sharpening factor, and generally 4, 8, or 16 times downsampling is used. No specific limit is made here. In another implementation, the high-resolution image is downsampled at at least two different downsampling ratios to form multi-ratio sample pairs; Specifically, the downsampling factor includes at least two of the following: integer factors 4×, 8×, 16× and non-integer factors 3×, 5×, 7×. A single training iteration covers integer / non-integer multiples, and during inference, only the interpolation size needs to be changed to output the corresponding high-resolution image, significantly improving deployment flexibility.

[0028] In this embodiment, after downsampling, the image is further normalized to scale the pixel values ​​to [-1,1] or [0,1]. Finally, the processed image data is divided into training set, test set and validation set.

[0029] S302: Construct the forward diffusion process, specifically, by gradually adding Gaussian noise to the high-resolution image through the forward noise-adding network until a pure noise image is obtained; S303: The low-resolution image obtained in step S301 is interpolated to the same size as the corresponding high-resolution image and used as a condition. It is then stitched together with the pure noise image at the current time step and input into the inverse denoising network to train the inverse denoising network to predict the added noise.

[0030] like Figure 3 As shown, the inverse denoising network adopts the U-Net noise prediction network architecture. The input data is processed through a downsampled convolutional residual module to obtain high-level features from the data. The DRB module includes group normalization, which can improve the model training efficiency and generalization ability. The convolutional layer is responsible for extracting high-level features from the data, and the LSTM layer extracts temporal features from the data. The attention mechanism makes the model pay more attention to important feature information, improving prediction accuracy. In this stage, the data length is reduced by 50% after downsampling, and the number of data channels is doubled after convolution. The data dimension is gradually restored through the upsampling convolutional residual module (URB). The URU module has the same network architecture as the DRB, except that it performs an upsampling operation at the end. The data length is doubled after the upsampling operation, and the number of data channels is reduced by 50% after the convolution operation, so that the final output data is consistent with the input data dimension. In this embodiment, by using a residual network to skip connections between the product residual modules of upsampling and downsampling, details in the downsampling process are preserved.

[0031] In this embodiment, a weighted combination of L1 loss, L2 loss and negative SSIM loss is used as the loss function for iterative training; The loss function is expressed as follows: .

[0032] S4: Obtain random noise image The image is then concatenated with the conditional image. The concatenated image is input into the inverse denoising network of the trained conditional diffusion model to predict the noise in the current noisy image. Based on the predicted noise, the next image is calculated. Then The process is repeated iteratively, using the input as the next step, until time step t=0, at which point a high-resolution image is output.

[0033] S5: Linearly maps the generated image pixel values ​​back to the dynamic range of the original image without additional sharpening or artifact removal.

[0034] Example 2 Embodiment 2 of the present invention discloses an image super-resolution enhancement processing device. The device can be a user equipment (UE) such as a mobile phone, smartphone, laptop computer, digital broadcast receiver, personal digital assistant (PDA), tablet computer (PAD), handheld device, in-vehicle device, wearable device, computing device, or other processing device connected to a wireless modem, or mobile station (MS), etc., used to perform the image super-resolution enhancement processing method. The device may be referred to as a user terminal, portable terminal, desktop terminal, etc.

[0035] Typically, the device includes: at least one processor, a memory, and an image super-resolution enhancement processing program stored in the memory and executable on the processor, the image super-resolution enhancement processing program being configured to implement the steps of the image super-resolution enhancement processing method as described in Example 1.

[0036] The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor can be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor may also include a main processor and coprocessors. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. The processor may also include an AI (Artificial Intelligence) processor, which handles computational operations related to the image super-resolution enhancement process, enabling the image super-resolution enhancement method to learn autonomously and improve efficiency and accuracy.

[0037] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory are used to store at least one instruction, which is executed by a processor to implement the image super-resolution sharpening processing method described in Embodiment 1.

[0038] In some embodiments, the terminal may also optionally include a communication interface and at least one peripheral device. The processor, memory, and communication interface can be connected via a bus or signal lines. Each peripheral device can be connected to the communication interface via a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit, a display screen, and a power supply.

[0039] The communication interface can be used to connect at least one I / O (Input / Output) related peripheral device to the processor and memory. The communication interface is used via the peripheral device to receive movement trajectories and other data uploaded by the user from multiple mobile terminals. In some embodiments, the processor, memory, and communication interface are integrated on the same chip or circuit board; in other embodiments, any one or two of the processor, memory, and communication interface can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0040] Radio frequency (RF) circuits are used to receive and transmit RF signals, also known as electromagnetic signals. RF circuits communicate with communication networks and other communication devices via electromagnetic signals, thereby acquiring the movement trajectories and other data of multiple mobile terminals. RF circuits convert electrical signals into electromagnetic signals for transmission, or convert received electromagnetic signals back into electrical signals. Optionally, RF circuits include: antenna systems, RF transceivers, one or more amplifiers, tuners, oscillators, digital signal processors, codec chipsets, user identity module cards, etc. RF circuits can communicate with other terminals through at least one wireless communication protocol. These wireless communication protocols include, but are not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit may also include circuitry related to NFC (Near Field Communication), which is not limited in this embodiment.

[0041] The display screen is used to display the UI (User Interface). This UI can include graphics, text, icons, videos, and any combination thereof. When the display screen is a touch screen, it also has the ability to collect touch signals on or above the surface of the display. These touch signals can be input as control signals to a processor for processing. In this case, the display screen can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, the display screen can be a single screen, the front panel of the electronic device; in other embodiments, there can be at least two screens, respectively disposed on different surfaces of the electronic device or in a folded design; in still other embodiments, the display screen can be a flexible screen, disposed on a curved or folded surface of the electronic device. Furthermore, the display screen can be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0042] A power supply is used to power the various components in an electronic device. The power supply can be alternating current (AC), direct current (DC), a disposable battery, or a rechargeable battery. When the power supply includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0043] Example 3 Embodiment 3 of the present invention discloses a computer storage medium, which is a readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the image super-resolution enhancement processing method described in Embodiment 1.

[0044] Specifically, the readable storage medium can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other readable storage medium capable of storing program code.

[0045] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. An image super-resolution enhancement method, characterized in that, include: S1: Obtain the low-resolution image to be processed and the target magnification; S2: Interpolate the low-resolution image to obtain the conditional image; S3: Construct and train a conditional diffusion model, which includes a forward noise-adding network and a backward noise-reducing network; S4: Obtain a random noise image and stitch it together with the conditional image. Input the stitched image into the inverse denoising network in the trained conditional diffusion model for iterative denoising and output a high-resolution image.

2. The image super-resolution enhancement method according to claim 1, characterized in that, In step S2, the interpolation process yields the conditional image, as detailed below: The low-resolution image is interpolated to the same initial pixel size as the target high-resolution image to obtain a conditional image.

3. The image super-resolution enhancement method according to claim 1, characterized in that, The reverse denoising network is a U-Net architecture, and an LSTM module is embedded in the jump connection path of the U-Net.

4. The image super-resolution enhancement method according to claim 3, characterized in that, In step S3, the training of the conditional diffusion model is specifically as follows: S301: Construct a high-resolution image training set and downsample the high-resolution images to obtain corresponding low-resolution images; S302: Gaussian noise is gradually added to the high-resolution image through the forward noise-adding network until a noisy image is obtained; S303: The low-resolution image obtained in step S301 is interpolated to the same size as the corresponding high-resolution image and used as a condition. It is then stitched together with the noisy image at the current time step and input into the inverse denoising network to train the inverse denoising network to predict the added noise.

5. The image super-resolution enhancement method according to claim 4, characterized in that, In step S301, the high-resolution image is downsampled at at least two different downsampling rates.

6. The image super-resolution enhancement method according to claim 4, characterized in that, In step S301, the downsampling factor includes at least two of the following: integer factors 4×, 8×, and 16×, and non-integer factors 3×, 5×, and 7×.

7. The image super-resolution enhancement method according to claim 4, characterized in that, In step S303, a weighted combination of L1 loss, L2 loss and negative SSIM loss is used as the loss function for iterative training; The loss function is expressed as follows: 。 8. The image super-resolution enhancement method according to any one of claims 1-7, characterized in that, After step S4, step S5 is also included: restoring the pixel values ​​of the generated high-resolution image to the dynamic range of the original image.

9. An image super-resolution enhancement processing device, characterized in that, The image super-resolution enhancement processing device includes: a memory, a processor, and an image super-resolution enhancement processing program stored in the memory and executable on the processor. When the image super-resolution enhancement processing program is executed by the processor, it implements the steps of the image super-resolution enhancement processing method according to any one of claims 1-8.

10. A computer storage medium, characterized in that, The storage medium stores an image super-resolution enhancement processing program, which, when executed by a processor, implements the steps of the image super-resolution enhancement processing method according to any one of claims 1-8.

Images