Image scaling method and system fusing high frequency information

Abstract

The application designs an end-to-end image scaling network structure, which learns the loss of high-frequency information distribution to strengthen feature learning and solves the ill-posed problem, thereby improving the image reconstruction effect. First, considering the ill-posed problem existing in image super-resolution, the high and low frequency information of the input image is separated to learn the high frequency information distribution in a more flexible way, and the learned information is randomly sampled and applied to image reconstruction to improve the image reconstruction effect. Secondly, a long skip connection is added to the up-sampling network to relieve the network depth, obtain faster convergence speed and better performance. Finally, a multi-stage training strategy is proposed for the optimization of the down-sampling and up-sampling sub-networks. A new effective image scaling method is constructed to provide a more efficient framework for image scaling in practical applications. Combined with the joint loss, the model loss is iteratively optimized to minimize the final loss of the model, thereby improving the accuracy and robustness of image reconstruction.

Description

Invention Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to an image scaling method and system that integrates high-frequency information. Background Technology

[0002] Image super-resolution, or super-resolution for short, refers to the process of restoring an image from low resolution to high resolution using algorithms. It is an important image processing technique in the fields of computer vision and image processing, and has significant applications in many areas, such as object detection, medical imaging, satellite remote sensing, and security monitoring. Because multiple high-resolution images always correspond to the same low-resolution image, the super-resolution problem is very challenging and is an ill-posed problem.

[0003] We categorize image-based super-resolution methods into three main types: interpolation-based methods, reconstruction and example-based methods, and deep learning-based methods. Reconstruction and example-based methods generally struggle to recover high-frequency detail information, leading to blurred reconstructed images, high computational cost, and poor real-time performance. Furthermore, convolutional network-based methods have achieved significant success in image super-resolution.

[0004] Image scaling is a fundamental and crucial technique in digital image processing, and a hot research topic in the field. It is also known as image resampling or resolution conversion. With the explosive growth of high-resolution images and videos on the internet, image downscaling is indispensable for storing, transmitting, and sharing such large amounts of data. High-resolution digital images are typically downscaled to fit various display screens or save storage costs while maintaining visually valid information. However, such downscaling inevitably places significant demands on the reverse engineering task: upscaling the downscaled image to a higher resolution or its original size, such as image super-resolution. How to balance the visual effect of low-resolution images with the quality of high-resolution images through algorithms, and what combination of image scaling algorithms can fully utilize the advantages of various methods, has become a key research focus.

[0005] According to the Nyquist-Shannon sampling theorem, high-frequency information is inevitably lost during image downscaling. Image super-resolution is inherently an ill-posed problem; due to its unfavorable nature, the quality of super-resolution results is very limited, and the lost high-frequency information cannot be well recovered. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, this invention provides a bidirectional image scaling learning method and system based on deep convolutional neural networks. Its purpose is to alleviate the ill-posed problem in image super-resolution, learn more effective information, and thus achieve better reconstructed image quality.

[0007] To achieve the above objectives, the present invention provides an image scaling method that integrates high-frequency information, comprising the following steps;

[0008] S1. Construct an image scaling network framework, which consists of two parts: a downsampling subnetwork and an upsampling subnetwork.

[0009] S2. Input the given input image into the downsampling sub-network to obtain a low-resolution image. At the same time, introduce latent variables to capture the distribution characteristics of high-frequency information, thereby learning the high-frequency information in the image downsampling process.

[0010] S3. The obtained low-resolution image, together with the randomly sampled high-frequency information, is input into the upsampling sub-network, and long skip connections are introduced during the upsampling process to perform feature aggregation, thereby obtaining the reconstructed image;

[0011] S4. Finally, the joint loss of the obtained features is calculated, which includes low-resolution image guidance loss, distribution matching loss and reconstruction loss.

[0012] This invention provides an end-to-end trainable neural network architecture and system for image scaling: It includes an image preprocessing unit for preprocessing a given input image, separating its information, and accurately separating high-frequency and low-frequency information; an image information learning unit for learning the separated high-frequency and low-frequency information separately; low-frequency information is used to generate a visually good low-resolution image through loss learning guided by a low-resolution image, while latent variables are introduced to capture the distribution characteristics of high-frequency information, thereby learning high-frequency information during image downsampling; and an image reconstruction unit that inputs the obtained low-resolution image, along with randomly sampled high-frequency information, into an upsampling network, and introduces long skip connections for feature aggregation during the upsampling process, finally obtaining a high-quality reconstructed image.

[0013] Furthermore, it also includes: a training unit for training the image scaling network model constructed by the image preprocessing unit, the image information learning unit, and the image reconstruction unit.

[0014] Furthermore, training the image scaling network model includes training the entire image scaling network model by minimizing the loss function.

[0015] Compared with existing technologies, it has the following beneficial effects:

[0016] This invention addresses the ill-posed problem in image super-resolution by introducing a Haar transform module into the downsampling network to decompose the input image into high- and low-frequency information. Furthermore, it learns the distribution of high-frequency information by introducing latent features. The learned information is then randomly sampled and fully utilized in the upsampling network reconstruction. Secondly, we propose a multi-stage training strategy that optimizes both downscale and upscale sub-networks to generate visually appealing low-resolution images. This effectively alleviates the ill-posed problem caused by the loss of high-frequency information during downsampling. Finally, we combine joint loss with feedback iterative optimization of the model's loss to minimize the final loss, thereby improving the accuracy and robustness of image reconstruction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a network structure diagram of image scaling that incorporates high-frequency information in this invention patent;

[0019] Figure 2 This is a structural diagram of the downsampling subnetwork in this invention patent;

[0020] Figure 3 This is a structural diagram of the upsampling subnetwork in this invention patent;

[0021] Figure 4 This is a reconstruction effect diagram of a specific embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. An image scaling network method for fusing high-frequency information includes steps S1 to S4:

[0024] S1. Construct an image scaling network framework, which consists of two parts: a downsampling subnetwork and an upsampling subnetwork.

[0025] S2. Input the given input image into the downsampling sub-network to obtain a low-resolution image. At the same time, introduce latent variables to capture the distribution characteristics of high-frequency information, thereby learning the high-frequency information in the image downsampling process.

[0026] S3. The obtained low-resolution image, together with the randomly sampled high-frequency information, is input into the upsampling sub-network, and long skip connections are introduced during the upsampling process to perform feature aggregation, thereby obtaining the reconstructed image;

[0027] S4. Finally, the joint loss of the obtained features is calculated, which includes low-resolution image guidance loss, distribution matching loss and reconstruction loss.

[0028] The following is a detailed description of each step.

[0029] In step S1, the network architecture is constructed, such as... Figure 1 As shown, the network consists of two parts: a downsampling subnetwork and an upsampling subnetwork. The specific steps are as follows:

[0030] S101. Construct a framework consisting of "DSNet + SRNet", which is composed of two sub-network modules;

[0031] (y down ,z)=F DS (x) (1)

[0032] x up =F SR (y down ,z) (2)

[0033] Where F DS (·) represents the downsampling network DSNet, F SR (·) represents the upsampling network SRNet. x is the given input image, y down The image is a low-resolution image obtained by a downsampling network. The network models the distribution of high-frequency information by introducing a latent variable z, which we force to be unknown and follow a specific Gaussian distribution, x. up It is a super-resolution image reconstructed by an upsampling network.

[0034] In step S2, the given input image is fed into the downsampling network to obtain a low-resolution image. Simultaneously, latent variables are used to capture the distribution characteristics of high-frequency information, thereby learning the high-frequency information during the image downsampling process. The specific steps are as follows:

[0035] S201, The framework diagram of the downsampling subnetwork is as follows: Figure 2As shown, the downsampling subnetwork consists of several downsampling modules, the number of which depends on the downsampling factor. Each downsampling module includes a Haar transform module and some stacked feature extraction modules. This subnetwork primarily learns the distribution of high-frequency information and produces visually appealing low-resolution images.

[0036] S202. First, the high-frequency and low-frequency information is effectively separated by the Haar transform module. This module has appeared in many classic network architectures and has been widely recognized and proven. Therefore, the Haar transform module is used to separate the high-frequency and low-frequency information of the input image.

[0037] S203. Next, the feature information obtained from the Haar transform module is input into the first feature extraction module. The feature extraction module consists of a series of dense modules, called the coupling layer. Its purpose is to further improve the high-frequency and low-frequency information to obtain a visually good low-resolution image and a potential representation of high-frequency information that follows a specific distribution.

[0038] {x L ,x H}=Haar(f i-1 (3)

[0039] f i =DB i (x L ,x H ), i = 1, 2, ..., k (4)

[0040] Where Haar(·) is the Haar transform module, DB i f represents the i-th feature extraction module. i-1 and f i represent the outputs of the (i-1)th and ith feature extraction modules, respectively, where f0 represents the input image given by the network.

[0041] In step S3, the obtained low-resolution image, together with randomly sampled high-frequency information, is input into the upsampling network.

[0042] S301, First, the framework diagram of the sampled subnetwork is as follows: Figure 3 As shown. This network mainly consists of an improved backbone network and long-skip connections. The network aims to utilize learned high-frequency information to reconstruct a better high-resolution image from an input low-resolution image.

[0043] S302. Next, we use EDSR as the backbone network, removing the preprocessing and post-processing operations (i.e., the MeanShift operation), and changing the network input. In addition to the provided low-resolution image, we extract information from a pre-specified distribution as additional input, obtaining the feature F of the EDSR backbone network. EDSR (y down,z);

[0044] S303, Finally, after processing the low-resolution image y down Perform bicubic linear interpolation F bic (·), then the magnified image x bic The super-resolution image is obtained by directly combining it with the backbone network output, as shown in the following formula:

[0045] x bic =F bic (y down (5)

[0046] x up =F EDSR (y down ,z)+x bic (6)

[0047] Where, x up The image is reconstructed by the upsampling network.

[0048] In step S4, the joint loss of the obtained features is calculated, including low-resolution image guidance loss, distribution matching loss, and reconstruction loss, and then fed back into the model for iterative loss optimization. The specific steps are as follows:

[0049] S401. The overall network loss consists of low-resolution image-guided loss, distribution matching loss, and reconstruction loss; following previous work, we treat each identity as a distinct category. The low-resolution image-guided loss is used to supervise the downsampling subnetwork during training, and is defined as follows:

[0050]

[0051] Where N is the total number of original high-resolution images in the training set. It's a real label. The image is a low-resolution image obtained from a downsampling subnetwork. The distance between the two images is calculated using L2 loss. In addition, a distribution matching loss is used to guide the learning of high-frequency information distribution.

[0052]

[0053] Where q(x) represents the sample cloud of the real high-resolution image data distribution, and p(z) represents the distribution of the introduced latent variable z. This represents the high-frequency information distribution of the downsampling subnetwork; finally, we minimize the expected difference between the reconstructed super-resolution image and the original ground truth image, and the reconstruction loss is defined as:

[0054]

[0055] Where, x (n) Represents the original, true image. The upsampling subnetwork reconstructs a high-resolution image, i.e. The distance between two images is calculated using L1 loss, and the total loss L total for:

[0056] L total =ω1×L down +ω2×L distr +ω3×L recon (14)

[0057] Where ω i This represents the weights used to balance different loss conditions.

[0058] S402. We employ a multi-stage training strategy to train the downsampling and upsampling sub-networks. First, we attempt to find a good initialization by training the downsampling sub-network separately. Using high-resolution images as input, we train the downsampling network separately using low-resolution image guidance loss and distribution matching loss to obtain high-quality low-resolution images and learn the distribution of high-frequency information. Then, we fix the parameters of the downsampling network and train the upsampling network using reconstruction loss. Finally, we jointly train the entire network module and fine-tune the parameters of the two sub-networks. Because the training process of the network model is a continuous optimization of the loss, we feed the currently obtained loss back into the network model for iterative optimization to reduce the loss and obtain more robust features. The results are as follows: Figure 4 As shown.

[0059] This invention designs an end-to-end image scaling network structure. By learning the distribution of lost high-frequency information to enhance feature learning, it alleviates the ill-posed problem and improves image reconstruction performance. First, considering the ill-posed problem in image super-resolution, high- and low-frequency information is separated from the input image to learn the high-frequency information distribution more flexibly. The learned information is then randomly sampled and applied to image reconstruction to improve its effectiveness. Second, a long skip connection is added to the upsampling network to reduce network depth, resulting in faster convergence and better performance. Finally, a multi-stage training strategy is proposed to optimize the downsampling and upsampling sub-networks. A novel and effective image scaling method is constructed, providing a more efficient framework for image scaling in practical applications. Furthermore, by combining joint loss and performing feedback iterative optimization of the model loss, the final loss of the model is minimized to improve the accuracy and robustness of image reconstruction.

[0060] This invention proposes an image scaling network that fuses high-frequency information, comprising:

[0061] A network framework is constructed, which consists of two parts: a downsampling subnetwork and an upsampling subnetwork.

[0062] The given input image is fed into the downsampling network to obtain a low-resolution image. At the same time, the distribution characteristics of high-frequency information are captured by latent variables, thereby learning the high-frequency information in the image downsampling process.

[0063] The obtained low-resolution image, together with randomly sampled high-frequency information, is input into the upsampling network, and long skip connections are introduced during the upsampling process to perform feature aggregation, thereby obtaining the reconstructed image;

[0064] Finally, the joint loss, low-resolution image guidance loss, distribution matching loss, and reconstruction loss are calculated from the obtained features.

[0065] On the other hand, an image scaling system that integrates high-frequency information is also provided, including an image preprocessing unit, an image information learning unit, and an image reconstruction unit. Wherein:

[0066] Image preprocessing unit: used to perform preprocessing operations on a given input image, to separate its information, and to accurately separate high-frequency and low-frequency information in the given image;

[0067] Image information learning unit: The high-frequency and low-frequency information are learned separately; the low-frequency information is used to generate a visually good low-resolution image by learning loss guided by the low-resolution image, while latent variables are introduced to capture the distribution characteristics of the high-frequency information, thereby learning the high-frequency information in the image downsampling process;

[0068] Image reconstruction unit: The obtained low-resolution image, together with the high-frequency information of random sampling, is input into the upsampling network. During the upsampling process, long skip connections are introduced to perform feature aggregation, and finally a high-quality reconstructed image is obtained.

[0069] This system is used to implement the functions of the methods in the above embodiments. Each module in the system corresponds to each step in the method, which has already been described in the method and will not be repeated here.

[0070] For example, it also includes a training unit for training the image scaling network model constructed by the image preprocessing unit, the image information learning unit, and the image reconstruction unit. Optionally, training the image scaling network model includes training the entire image scaling network model by minimizing a loss function.

[0071] This embodiment separates high- and low-frequency information in the input image, randomly samples the learned high-frequency information, and applies it fully and reasonably to the image reconstruction process, thereby improving the image reconstruction effect. Furthermore, a long skip connection is added to the upsampling network to alleviate the network depth, resulting in faster convergence speed and better performance, providing a more efficient framework for image scaling in practical applications.

[0072] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. An image scaling method that integrates high-frequency information, characterized in that, The method is executed by a computer and includes the following steps: S1. Construct an image scaling network framework, which consists of two parts: a downsampling subnetwork and an upsampling subnetwork; the specific implementation process is as follows: S101. Construct a framework consisting of "DSNet + SRNet", which is composed of two sub-network modules: upsampling and downsampling. (1) (2) in It is the downsampling network DSNet. It is the upsampling network SRNet. Given an input image, It is a low-resolution image obtained by a downsampling network, and the network introduces latent variables. To model the distribution of high-frequency information, we force it to be unknowable and follow a specific Gaussian distribution. It is a super-resolution image reconstructed by an upsampling network; S2. The given input image is fed into the downsampling sub-network to obtain a low-resolution image. At the same time, latent variables are introduced to capture the distribution characteristics of high-frequency information, thereby learning the high-frequency information in the image downsampling process. The specific implementation process is as follows: S201. This downsampling subnetwork consists of several downsampling modules, the number of which depends on the downsampling factor. Each downsampling module includes a Haar transform module and some stacked feature extraction modules. This subnetwork mainly learns the distribution of high-frequency information and produces visually appealing low-resolution images. S202. First, the high-frequency and low-frequency information is effectively separated by the Haar transform module. This module appears in many classic network architectures and has been widely recognized and proven. Therefore, the Haar transform module is used to separate the high-frequency and low-frequency information of the input image. S203. Next, the feature information obtained from the Haar transform module is input into the first feature extraction module. The feature extraction module consists of a series of dense modules, called the coupling layer. Its purpose is to further improve the high-frequency and low-frequency information to obtain a visually good low-resolution image and a potential representation of high-frequency information that follows a specific distribution. (3) (4) in It is a Haar converter module. Indicates the first Block feature extraction module and These represent the outputs of the feature extraction modules for the (i-1)th and i-th blocks, respectively. This represents the input image provided by the network; S3. The obtained low-resolution image, together with the randomly sampled high-frequency information, is input into the upsampling sub-network, and long skip connections are introduced during the upsampling process to perform feature aggregation, thereby obtaining the reconstructed image; S4. Finally, the joint loss of the obtained features is calculated, which includes low-resolution image guidance loss, distribution matching loss and reconstruction loss.

2. The image scaling method fusing high-frequency information as described in claim 1, characterized in that, The specific implementation process of upsampling subnetwork feature aggregation and high-resolution image reconstruction, corresponding to S3, is as follows: S301. First, the upsampling subnetwork mainly consists of an improved backbone network and long skip connections; this network aims to use the learned high-frequency information to reconstruct the input low-resolution image into a better high-resolution image. S302. Next, we use EDSR as the backbone network, removing the preprocessing and post-processing operations (i.e., the MeanShift operation), and changing the network input. In addition to the provided low-resolution image, we extract information from a pre-specified distribution as additional input, obtaining the features of the EDSR backbone network. ; S303, finally, the low-resolution image is processed again. Perform bicubic interpolation Then enlarge the image The super-resolution image is obtained by directly combining it with the backbone network output, as shown in the following formula: (5) (6) in, The image is reconstructed by the upsampling network.

3. The image scaling method incorporating high-frequency information as described in claim 1, characterized in that, The specific implementation process of calculating the joint loss to train the image scaling network, corresponding to S4, is as follows: S401. The overall network loss consists of low-resolution image-guided loss, distribution matching loss, and reconstruction loss. Following previous work, we treat each identity as a distinct category. The low-resolution image-guided loss is used to supervise the downsampling subnetwork during training. The low-resolution image-guided loss is defined as follows: (7) Where N is the total number of original high-resolution images in the training set. It's a real label. It is a low-resolution image obtained by a downsampling subnetwork, utilizing The loss is calculated by determining the distance between two images; additionally, a distribution matching loss is used to guide the learning of high-frequency information distributions. (8) in, A sample cloud representing the distribution of real high-resolution image data. Indicates the introduced latent variables The distribution that it follows This represents the high-frequency information distribution of the downsampling subnetwork; finally, we minimize the expected difference between the reconstructed super-resolution image and the original ground truth image, and the reconstruction loss is defined as: (9) in, Represents the original, true image. The upsampling subnetwork reconstructs the high-resolution image, i.e. ,use The loss is calculated based on the distance between the two images; the total loss is calculated as follows: for: + + (10) in This represents the weights used to balance different loss conditions; S402. We employ a multi-stage training strategy to train the downsampling and upsampling sub-networks. First, we attempt to find a good initialization by training the downsampling sub-network separately. Using high-resolution images as input, we train the downsampling network separately using low-resolution image guidance loss and distribution matching loss to obtain high-quality low-resolution images and learn the distribution of high-frequency information. Then, we fix the parameters of the downsampling network and train the upsampling network using reconstruction loss. Finally, we jointly train the entire network module and fine-tune the parameters of the two sub-networks. Because the training process of the network model is a process of continuously optimizing the loss, we feed the currently obtained loss back into the network model for continuous iterative optimization to reduce the loss and thus obtain more robust features.

4. An image scaling system that integrates high-frequency information, characterized in that, Performing the steps of any one of claims 1-3 includes: Image preprocessing unit: used to perform preprocessing operations on a given input image, to separate its information, and to accurately separate high-frequency and low-frequency information in the given image; Image information learning unit: The high-frequency and low-frequency information are learned separately; the low-frequency information is used to generate a visually good low-resolution image by learning loss guided by the low-resolution image, while latent variables are introduced to capture the distribution characteristics of the high-frequency information, thereby learning the high-frequency information in the image downsampling process; Image reconstruction unit: The obtained low-resolution image, together with the high-frequency information of random sampling, is input into the upsampling network. Long skip connections are introduced during the upsampling process to perform feature aggregation, and finally a high-quality reconstructed image is obtained.

5. The system according to claim 4, characterized in that, Also includes: The training unit is used to train the image scaling network model constructed by the image preprocessing unit, the image information learning unit, and the image reconstruction unit.

6. The system according to claim 4, characterized in that, Training the image scaling network model includes: The entire image scaling network model is trained by minimizing the loss function.

Images