Image super-resolution method, system and equipment based on low-resolution increment

Abstract

The invention relates to an image super-resolution method, system and equipment based on low-resolution increment, and the method comprises the steps: carrying out the initial down-sampling and multi-round increment reconstruction, obtaining information increment through the additional down-sampling of each round of increment reconstruction, and carrying out the up-sampling and super-resolution processing based on the information increment until a reconstructed image meeting the requirements is obtained. Through multi-round on-demand additional down-sampling and focusing on an interested area (which can be a whole image or one or more local image blocks) of a user, incremental optimization is realized, transmission of redundant data is effectively reduced, unnecessary resolution overhead is avoided, dependence on low-resolution image size is reduced, data transmission and processing are enabled to be more simplified, and user experience is improved. The system efficiency is improved, and the bandwidth is saved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image super-resolution method, system and device based on low-resolution increments. Background Art

[0002] The continuous improvement of camera resolution has brought people a better visual experience. However, even with the use of high-performance compression algorithms, transmitting a large number of large-size, high-resolution images on the Internet will still put tremendous pressure on the core network and access network. In fact, in many scenarios, due to the limited screen size of display devices (such as smartphones), images with original resolution are not required. Image super-resolution methods effectively solve this problem: the sender can downsample the original image and transmit its low-resolution version to match the size of the target display device; if necessary, the receiver can restore the image to the original resolution size through image super-resolution (SR) technology.

[0003] There are many well-designed models for image super-resolution (SR), which focus on improving the quality of super-resolution images. However, the quality of super-resolution images is heavily dependent on and constrained by the super-resolution magnification scale, that is, the closer the size of the low-resolution image is to the original image, the higher the quality of the super-resolution image. In other words, when the quality of the super-resolution image cannot meet user needs, it can be improved by retransmitting a larger low-resolution image version, but this will lead to redundant transmission and bandwidth waste.

[0004] That is, existing image super-resolution methods have technical problems of relying on low-resolution image size, redundant transmission and bandwidth waste. Summary of the invention

[0005] Based on this, it is necessary to provide an image super-resolution method based on low-resolution increments, an image super-resolution system based on low-resolution increments, and a computer device to address the above technical problems.

[0006] In order to achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0007] On the one hand, a method for image super-resolution based on low-resolution increment is provided, comprising:

[0008] Perform initial downsampling on the original image to obtain initial downsampling information, perform upsampling and super-resolution processing based on the initial downsampling information to obtain a restored image; downsampling refers to the process of reducing the resolution of a high-resolution image, and upsampling refers to the process of increasing the resolution of a low-resolution image;

[0009] In the current incremental reconstruction round, additional downsampling is performed based on an on-demand mechanism to obtain information increment, and upsampling and super-resolution processing are performed based on the information increment and the initial downsampling information to obtain a reconstructed image;

[0010] Additional downsampling is performed based on an on-demand mechanism to obtain the next information increment, and the reconstruction of the next incremental reconstruction round is started until the final reconstructed image meets the requirements, at which time the reconstruction is stopped.

[0011] In one embodiment, the image super-resolution method based on low-resolution increments further includes: an initial downsampling process is:

[0012]

[0013] Among them, p k represents the kth downsampled low-resolution pixel, denotes a convolution operation, φ(k) denotes the kth channel of the convolution kernel φ, Q(·) denotes a quantization module, and X denotes an H×W image.

[0014] In one embodiment, the image super-resolution method based on low-resolution increments, the on-demand mechanism includes:

[0015] According to the visual effect of the reconstructed image or the no-reference index, determine whether the reconstructed image meets the requirements;

[0016] When it is determined that the requirements are not met, the image area to be supplemented and its corresponding sampling increment are determined, and a request is sent to the sender;

[0017] The sender performs additional downsampling based on the request and obtains information increment using a block-based downsampling strategy.

[0018] In one embodiment, in the image super-resolution method based on low-resolution increment, the initial downsampling information is:

[0019]

[0020] Among them, α0 represents the initial downsampling information, J represents that the original image is divided into J blocks, Indicates the number of low-resolution pixels contained in the initial downsampling information of the jth block of the original image, represents the kth downsampled low-resolution pixel of the jth block of the original image, and B j Represented as the jth block of the original image.

[0021] In one embodiment, in the image super-resolution method based on low-resolution increment, the information increment is:

[0022]

[0023] Among them, α i represents the information increment of the ith additional downsampling, where Represents the total number of low-resolution pixels accumulated in the i-1th round of the jth block of the original image, Represents the number of low-resolution pixels contained in the i-th round of the sampling increment of the j-th block of the original image, wherein the convolution kernel is initially designed to have n channels, and the sampling increment is obtained by expanding m channels.

[0024] In one embodiment, the image super-resolution method based on low-resolution increment, the upsampling process includes:

[0025]

[0026] Where LR0 represents the visual form information obtained in the initial downsampling; R(·) represents the rearrangement operation; LR i represents the visual form information obtained in the i-th incremental reconstruction round; C(·,·) represents the connection operation, conv(·) represents the convolutional layer; upsampling and the second rearrangement operation are performed according to the visual form information.

[0027] In one embodiment, the image super-resolution method based on low-resolution increment, before the step of initially downsampling the original image, further includes:

[0028] The image super-resolution model is trained. The training process includes:

[0029] Jointly train the initial downsampling module, the initial upsampling module, and the super-resolution module, and optimize them through the loss function until convergence, then fix the parameters of the initial downsampling module;

[0030] Continue to train the additional downsampling module, the additional upsampling module and the super-resolution module, and optimize the loss function until convergence to obtain a trained image super-resolution model.

[0031] In one embodiment, the image super-resolution method based on low-resolution increment, the loss function is:

[0032]

[0033] Among them, mse represents mean square error, l1 represents standard l1 norm loss, LR i represents the visual form information obtained in the i-th incremental reconstruction round, and LR represents LR i The real reference, SR i represents the reconstructed image obtained in the i-th incremental reconstruction round, and HR represents the original image.

[0034] On the other hand, a low-resolution increment-based image super-resolution system is also provided, comprising:

[0035] An initial downsampling module is used to perform initial downsampling on the original image to obtain initial downsampling information;

[0036] The initial upsampling module and the super-resolution module are used to perform upsampling and super-resolution processing according to the initial downsampling information to obtain a restored image; downsampling refers to the process of reducing the resolution of a high-resolution image, and upsampling refers to the process of increasing the resolution of a low-resolution image;

[0037] An additional downsampling module is used to perform additional downsampling based on an on-demand mechanism in the current incremental reconstruction round to obtain information increment;

[0038] An additional upsampling module and a super-resolution module are used to perform upsampling and super-resolution processing according to the information increment and the initial downsampling information to obtain a reconstructed image;

[0039] The additional downsampling module, the additional upsampling module and the super-resolution module are also used to perform additional downsampling based on an on-demand mechanism to obtain the next information increment and enter the reconstruction of the next incremental reconstruction round until the final reconstructed image meets the requirements and the reconstruction is stopped.

[0040] On the other hand, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned image super-resolution method based on low-resolution increments when executing the computer program.

[0041] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0042] The above-mentioned image super-resolution method, system and device based on low-resolution increments, through initial downsampling and multiple rounds of incremental reconstruction, each round of incremental reconstruction relies on additional downsampling to obtain information increments, and then upsampling and super-resolution processing are performed based on the information increment until a reconstructed image that meets the requirements is obtained. Through multiple rounds of additional downsampling on demand, the area of ​​interest to the user (which can be the entire image or one or more local image blocks) is focused to achieve incremental optimization, effectively reduce the transmission of redundant data, avoid unnecessary resolution overhead, reduce dependence on low-resolution image size, ensure that data transmission and processing are more streamlined, improve system efficiency, and save bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A schematic diagram of a process of an image super-resolution method based on low-resolution increments in one embodiment;

[0045] Figure 2 A schematic diagram of an image super-resolution framework based on low-resolution increments in one embodiment;

[0046] Figure 3 Schematic diagram of a block-based convolution structure in one embodiment;

[0047] Figure 4 Schematic diagram of comparison of super-resolution image transmission costs based on SSIM in one embodiment, wherein (a) represents the comparison result obtained using the data set Set5, (b) represents the comparison result obtained using the data set Set14, (c) represents the comparison result obtained using the data set BSDS100, (d) represents the comparison result obtained using the data set Urban100, and (e) represents the comparison result obtained using the data set DIV2K;

[0048] Figure 5 Schematic diagram of comparison of super-resolution image transmission cost based on PSNR in one embodiment, wherein (a) represents the comparison result obtained using the data set Set5, (b) represents the comparison result obtained using the data set Set14, (c) represents the comparison result obtained using the data set BSDS100, (d) represents the comparison result obtained using the data set Urban100, and (e) represents the comparison result obtained using the data set DIV2K;

[0049] Figure 6 A schematic diagram of the impact of block-based local information increment on transmission cost in one embodiment;

[0050] Figure 7 A schematic diagram of the effect of gradually adding incremental information on the visual effect of a super-resolution image in one embodiment;

[0051] Figure 8 FIG. 1 is a schematic diagram of the influence of block-based local information increment on super-resolution images in one embodiment, wherein: Figure 8 (a) represents the 0831st low-resolution image in DIV2k. Figure 8 (b) Super-resolution image showing the block-based local information increment;

[0052] Fig. 9 Schematic diagram of the impact of unified and separate training strategies on super-resolution performance in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0055] It should be noted that the reference to "embodiment" in this document means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The presentation of the phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments.

[0056] The following will describe the implementation of the present invention in detail with reference to the accompanying drawings in the embodiment diagram of the present invention.

[0057] In one embodiment, Figure 1 As shown, the embodiment of the present application provides an image super-resolution method based on low-resolution increments, and the method includes the following processing steps S12-16:

[0058] S12, performing initial downsampling on the original image to obtain initial downsampling information, performing upsampling and super-resolution processing based on the initial downsampling information to obtain a restored image; downsampling refers to the process of reducing the resolution of a high-resolution image, and upsampling refers to the process of increasing the resolution of a low-resolution image.

[0059] It can be understood that the initial downsampling is to reduce the resolution of the high-resolution original image and extract the initial downsampling information. After that, the resolution of the low-resolution image is improved by upsampling and super-resolution processing using the initial downsampling information to generate a restored image.

[0060] Specifically, downsampling can be achieved through mean pooling, bilinear interpolation, or convolution operations. Upsampling can use deconvolution, upsampling layers, or sub-pixel convolution operations to increase the resolution of downsampled information to the target resolution. Through super-resolution network models, such as SRCNN (super-resolution convolutional neural network), the upsampled image is further optimized to generate a restored image. The network can also combine adversarial learning, perceptual loss, or self-supervised learning to improve the clarity and authenticity of the image.

[0061] S14, in the current incremental reconstruction round, additional downsampling is performed based on an on-demand mechanism to obtain information increment, and upsampling and super-resolution processing are performed according to the information increment and the initial downsampling information to obtain a reconstructed image.

[0062] It can be understood that in the current incremental reconstruction process, the high-resolution original image is additionally downsampled as needed to generate low-resolution reference data called information increment, and upsampling is performed based on the information increment and the initial downsampling information, and finally the super-resolution algorithm is used for further optimization to generate a new reconstructed image.

[0063] Specifically, based on on-demand downsampling, downsampling can be performed based on a specific area to obtain additional supplementary information as information increment. This specific area is the area of ​​interest to the user (it can be the entire image or one or more local image blocks). The information increment and the initial downsampling information are fused, such as by weighting or splicing, and then up-sampled and super-resolution processed to output a high-quality reconstructed image.

[0064] S16, performing additional downsampling based on an on-demand mechanism to obtain the next information increment, and entering the reconstruction of the next incremental reconstruction round until the final reconstructed image meets the requirements, and then stopping the reconstruction.

[0065] It can be understood that the next information increment is obtained and the reconstruction of the next incremental reconstruction round is entered. Specifically, incremental reconstruction is performed based on the previously obtained information increment, the initial downsampling information and the next information increment obtained. Repeat multiple rounds of incremental reconstruction until the final reconstructed image meets the requirements, and then stop the reconstruction. Specifically, it can be judged based on the visual effect whether the requirements are met, or it can be evaluated by a non-reference indicator such as PI. The image area of ​​interest to the user is used as the optimization target (it can be the entire image or one or more local image blocks) to ensure that the reconstruction result can meet the actual needs.

[0066] In the above-mentioned image super-resolution method based on low-resolution increments, initial downsampling and multiple rounds of incremental reconstruction are performed, and each round of incremental reconstruction relies on additional downsampling to obtain information increments, which are then processed by upsampling and super-resolution based on the information increments until a reconstructed image that meets the requirements is obtained. Through multiple rounds of additional downsampling on demand, the user's area of ​​interest (which can be the entire image or one or more local image blocks) is focused to achieve incremental optimization, effectively reduce the transmission of redundant data, avoid unnecessary resolution overhead, reduce dependence on low-resolution image size, ensure more streamlined data transmission and processing, improve system efficiency, and save bandwidth.

[0067] In one embodiment, the initial downsampling process is:

[0068]

[0069] Among them, p k represents the kth down-sampled low-resolution pixel, denotes a convolution operation, φ(k) denotes the kth channel of the convolution kernel φ, Q(·) denotes a quantization module, and X denotes an H×W image.

[0070] It can be understood that the image super-resolution framework based on low-resolution increments is as follows Figure 2 As shown, it includes an initial downsampling module D-S0 and an additional downsampling module DS i (i≥1, the same understanding will be given later, no further explanation), initial upsampling module U-S0, additional upsampling module US i The super-resolution module (i.e., SR module) and the downsampling module are executed on the sending side.

[0071] Initial downsampling, random parameters of the convolution kernel φ can be used to sample the image, the convolution kernel The size can be designed to be H×W, matching the height and width of the image to be transmitted. In this application, the block-based convolution is regarded as retaining the parameters within the h×w block and setting the remaining parameters to 0, that is, in essence, the block-based convolution still samples the entire image.

[0072] Among them, Q(·) is a quantization module, which is used to convert the original floating point number (usually a high-precision 32-bit or 16-bit floating point number) into an 8-bit unsigned integer through quantization processing, because images are usually represented by integers between 0 and 255.

[0073] At the beginning of image X transmission, in order to save bandwidth, the sender first transmits the initial downsampling information α0 of the image. Assuming that α0 contains n0 low-resolution pixels, α0 can be expressed as:

[0074]

[0075] If the sender receives m i If a request for additional pixels is made, module DS is executed. i , and add m i low-resolution pixels, using information increment α i express:

[0076]

[0077] in, The process of sending bits in response to requests is repeated until the user is satisfied. i They represent the initial downsampling information and the i-th additional downsampling information increment respectively.

[0078] In one embodiment, the image super-resolution method based on low-resolution increments, the on-demand mechanism includes:

[0079] According to the visual effect of the reconstructed image or the no-reference index, it is judged whether the reconstructed image meets the requirements; when it is judged that it does not meet the requirements, the image area to be supplemented (which can be the whole image or one or more local image blocks) and its corresponding sampling increment are determined, and a request is sent to the sender; according to the request, the sender adopts a block-based downsampling strategy to perform additional downsampling to obtain information increment.

[0080] It can be understood that the visual effect of the reconstructed image can be used to determine whether the reconstructed image meets the quality requirements, or it can be determined by a non-reference indicator; if not, the image area to be supplemented and its sampling increment are determined, and a request is sent to the sender. The sender performs additional downsampling based on the block-level downsampling strategy to generate information increments for supplementary reconstruction. In the specific implementation, the system uses an iterative method to gradually optimize the reconstruction effect for the area of ​​interest (which can be the entire image or one or more local image blocks) to achieve accurate data acquisition. This method reduces unnecessary data transmission, optimizes data processing efficiency, ensures the quality of the reconstructed image, and reduces transmission costs. In addition, the use of non-reference indicators does not rely on the original image, and can directly evaluate the quality from the reconstructed image. It is suitable for scenarios with unknown original data, thereby improving the universality and real-time nature of the evaluation.

[0081] In one embodiment, based on the block downsampling strategy, the initial downsampling information is:

[0082]

[0083] Among them, α0 represents the initial downsampling information, J represents that the original image is divided into J blocks, Indicates the number of low-resolution pixels contained in the initial downsampling information of the jth block of the original image, represents the kth downsampled low-resolution pixel of the jth block of the original image, and B j Represented as the jth block of the original image.

[0084] It can be understood that the block-based convolutional structure is Figure 3 As shown, the size φ(k) of the convolution kernel on the kth channel is designed to be h×w, the block size of the original image is H×W, and the original image is divided into J blocks, where J=rs, r=H / h, s=W / w.

[0085] The information increment is:

[0086]

[0087] Among them, α i represents the information increment of the ith additional downsampling, where Represents the total number of low-resolution pixels accumulated in the i-1th round of the jth block of the original image, Represents the number of low-resolution pixels contained in the i-th round of the sampling increment of the j-th block of the original image, wherein the convolution kernel is initially designed to have n channels, and the sampling increment is obtained by expanding m channels.

[0088] Among them, the block-based convolution can be regarded as retaining the parameters within the h×w block and setting the rest of the parameters to 0. In essence, the block-based convolution still samples the entire image. However, by setting multiple parameters to 0, the block-based strategy avoids the convolution operation with a very large number of parameters, making the downsampling process feasible.

[0089] The above-mentioned image super-resolution method based on low-resolution increments uses convolution and its random parameters to generate downsampled low-resolution information, where the n channels of the convolution kernel correspond to n downsamplings. Since an additional m low-resolution pixels are required, the convolution kernel further expands the m channels to provide the ability to add low-resolution information increments at any scale. The information increments of multiple iterations are then gradually integrated through a carefully designed upsampling module, and the SR module performs super-resolution processing. Through block division, integration can be performed block by block to achieve on-demand super-resolution.

[0090] In one embodiment, the upsampling process includes:

[0091]

[0092] Where LR0 represents the visual form information obtained in the initial downsampling; R(·) represents the rearrangement operation; LR irepresents the visual form information obtained in the i-th incremental reconstruction round; C(·,·) represents the connection operation, conv(·) represents the convolutional layer; upsampling and the second rearrangement operation are performed according to the visual form information.

[0093] Understandably, combined Figure 2 As shown in Figure 1, the upsampling process includes two rearrangement operations, a concatenation operation, and an upsampling operation. In the rearrangement operation, α0 is rearranged into its visual form LR0. If additional information increment α1 is available, the features of LR0 are concatenated with the new available information increment α1 through the convolutional layer conv(·), and the result is rearranged to generate LR1. This concatenation and rearrangement process can be repeated until the user's needs are met. Among them, LR i is the corresponding visual version after the i-th increment.

[0094] Specifically, the upsampling operation is responsible for converting a small-sized low-resolution version of image X to LR i Upsample to a suitable scale. Since the existing super-resolution (SR) module only supports upsampling at a fixed ratio (such as integer multiples), and the method of this application supports upsampling and downsampling at any ratio, before performing super-resolution processing, the LR can be upsampled to a suitable scale through the upsampling module. i Adjust to the appropriate scale and then input into the SR module.

[0095] Upsampling is the inverse process of downsampling and is also achieved by using a convolutional structure. Let N = H × W. Then, the size of the convolution kernel Φ(k) for the i-th upsampling is n i , the number of channels is N. The output of N convolution operations is:

[0096]

[0097] in These N pixels are rearranged into a size of H×W and used as the input of the SR module. After super-resolution processing by the SR module, SR0 and SR i , SR0 represents the restored image, SR i represents the reconstructed image of the i-th incremental reconstruction round.

[0098] For block-based upsampling, the convolution layer conv(·) and the connection operation C(·,·) are also based on image blocks, while the second rearrangement operation R(·) further arranges the single block into a complete image, which is then input into the SR module. Through the processing of the SR module, the block artifacts can be improved.

[0099] In one embodiment, before the step of initially downsampling the original image, the method further includes: training an image super-resolution model, wherein the training process includes:

[0100] The initial downsampling module, the initial upsampling module and the super-resolution module are jointly trained and optimized through the loss function until convergence, and the parameters of the initial downsampling module are fixed; the additional downsampling module, the additional upsampling module and the super-resolution module are continued to be trained and optimized through the loss function until convergence to obtain a trained image super-resolution model.

[0101] It can be understood that the image super-resolution model is first trained in stages. In the initial stage, the initial downsampling module, the initial upsampling module and the super-resolution module are jointly trained, and the loss function is used to optimize them until the model converges. In this stage, the parameters of the initial downsampling module are continuously adjusted to achieve the best downsampling effect. Subsequently, the parameters of the initial downsampling module are fixed, and the second stage is entered to continue multiple rounds of training of the additional downsampling module, the additional upsampling module and the super-resolution module to further optimize the overall model. Through this staged training method, the initial downsampling module can stably provide basic image information, and the additional module can further improve the quality of the entire image on this basis. Finally, the trained image super-resolution model is obtained through the optimization of the loss function.

[0102] The above-mentioned image super-resolution method based on low-resolution increments effectively improves the stability and efficiency of model training in a phased training method. In the initial stage, by optimizing the parameters of the initial downsampling module, the effective retention of image information during the downsampling process is ensured. Afterwards, on the basis of fixing the parameters of the initial downsampling module, the remaining modules are further trained to improve the quality of the entire image. It can achieve more accurate image restoration effects and improve the generalization ability and application effect of the model.

[0103] In one embodiment, the loss function of the image super-resolution method based on low-resolution increment is:

[0104]

[0105] Among them, mse represents mean square error, l1 represents standard l1 norm loss, LR i represents the visual form information obtained in the i-th incremental reconstruction round, and LR represents LR i The real reference, HR represents the original image, SR i Denotes the reconstructed image obtained in the i-th incremental reconstruction round.

[0106] It can be understood that in addition to using the above loss functions, more complex feature-level perceptual losses can also be used to improve performance, thereby not only focusing on the pixel differences of the image, but also taking into account the perceptual quality and high-level features of the image (such as edges, textures, structures, etc.). Specifically, feature-level perceptual losses are usually achieved by using pre-trained deep neural networks, such as the VGG (Visual Geometry Group) network, to extract high-level features of the image. Certain layers in the VGG network can be used to extract intermediate features of the image (such as the output of the convolutional layer), and then the differences between the reconstructed image and the original image in these features are calculated, rather than relying solely on pixel differences. This is closer to the way the human visual system perceives things, and can therefore provide a more natural and delicate image restoration effect.

[0107] In some embodiments, in order to more intuitively and comprehensively illustrate the above-mentioned image super-resolution method based on low-resolution increments, the following is an application example of the image super-resolution method based on low-resolution increments. It should be noted that the implementation cases given in this specification are only illustrative and are not the only limitation on the specific implementation cases of the present invention. Those skilled in the art can adopt the above-mentioned image super-resolution method based on low-resolution increments under the guidance of the implementation cases provided by the present invention, reduce the dependence on the size of the low-resolution image, and focus on the area of ​​interest of the user, so as to achieve incremental optimization and effectively reduce the transmission of redundant data.

[0108] The latest transformer-based SR network Omni is adopted as the SR module. It is worth noting that any other SR network can also be used as a SR module in a similar way.

[0109] 1. Experimental setup:

[0110] Datasets and metrics. DIV2K and Flickr2K are used as training datasets. DIV2K provides 1,000 images with resolutions up to 2K, while Flickr2K contains 2,650 high-quality images collected from the Flickr platform. These datasets support the model to achieve excellent performance in super-resolution tasks through rich image details and diversity. In order to speed up image loading during training, each training image is evenly divided into four parts, each of which becomes an independent, relatively small training image.

[0111] For testing purposes, we used 5 standard benchmark datasets, including Set5, Set14, BSDS100, Urban100, and DIV2K validation sets. Set5 and Set14 provide classic low-resolution-high-resolution image pairs, which are suitable for preliminary verification of model effects; BSDS100 contains more complex natural scene images to test the model's performance on diverse content; Urban100 focuses on urban buildings and structural textures, challenging the model's ability to restore details; and the DIV2K validation set provides a modern benchmark for high-resolution images to measure the model's applicability in high-resolution scenes. Through these diverse datasets, the performance of the model in different scenarios can be comprehensively evaluated.

[0112] PSNR (peak signal-to-noise ratio) and SSIM (structural similarity) are used to evaluate the quality of the super-resolution image on the Y channel of the transformed YCbCr space (brightness and color difference space).

[0113] Implementation details: The training images are augmented by random flipping and 90 degree rotation, and the ground truth (true reference) of ×4 low resolution is generated from the corresponding original image by bicubic interpolation algorithm. The proposed framework is trained with AdamW optimizer (weight decay adaptive momentum optimizer) with batch size of 16 and 300K iterations. The initial learning rate is set to 0.0005 and halved every 50K iterations. In each training batch, the original high-resolution image is randomly cropped into small patches of size 64×64 as input. All experiments are performed using PyTorch2.2.0 on a computer equipped with NVIDIA GeForce RTX 4090 GPU, Intel Core i7-14700KF clocked at 3.40GHz and 64GB of memory.

[0114] Block size. The downsampling module adopts a block-based convolutional structure. The block sizes of the original image and its ×4 low-resolution version are set to 32×32 and 8×8, respectively.

[0115] 2. Transmission cost:

[0116] The transmission cost is estimated using the ratio of the downsampled information increment to the original image. Scenario 1: SR-based transmission method: If the super-resolution image does not meet the requirements, a larger low-resolution image version must be retransmitted. Scenario 2: Using the method of this application, additional downsampling is performed based on an on-demand mechanism. Specifically, a block-based convolutional structure is adopted. For fair comparison, both scenarios 1 and 2 use Omni.

[0117] Figure 4The figure is a schematic diagram of the comparison of super-resolution image transmission costs based on SSIM, where (a) represents the comparison results obtained using the dataset Set5, (b) represents the comparison results obtained using the dataset Set14, (c) represents the comparison results obtained using the dataset BSDS100, (d) represents the comparison results obtained using the dataset Urban100, and (e) represents the comparison results obtained using the dataset DIV2K. Omni-based transmission represents the transmission cost and quality based on the SR transmission method in scenario 1, and Our progressive approach represents the transmission cost and quality based on the method of this application in scenario 2. The transmission cost is calculated as the percentage of the information increment to the original image, and the transmission quality is represented by SSIM. Proportion represents the percentage.

[0118] Figure 5 The figure is a schematic diagram of the comparison of super-resolution image transmission costs based on PSNR, where (a) represents the comparison results obtained using the dataset Set5, (b) represents the comparison results obtained using the dataset Set14, (c) represents the comparison results obtained using the dataset BSDS100, (d) represents the comparison results obtained using the dataset Urban100, and (e) represents the comparison results obtained using the dataset DIV2K. General SR-based transmission also represents the transmission cost based on the SR transmission method in scenario 1, and Our progressive SR-based transmission also represents the transmission cost based on the method of this application in scenario 2. The transmission cost is calculated as the percentage of the information increment to the original image, and the transmission quality is represented by PSNR. Data size represents the data size (percentage).

[0119] Figure 4-Figure 5 In the figure, for each benchmark dataset, the first point from the left represents the transmission cost α0 of the initial low-resolution version. At this time, the methods of scenario 1 and scenario 2 are the same in transmission cost. However, when the quality of the super-resolution image corresponding to α0 is insufficient, the method of scenario 1 needs to retransmit a larger low-resolution image of size α0+α1, while the method of scenario 2 (this application) only adds an increment α1. The results show that under the same super-resolution image quality, the transmission cost of the method of this application is significantly lower; in addition, as the image quality gradually improves, the savings in transmission cost of the method of this application are more obvious.

[0120] Figure 6The figure shows the effect of block-based local information increment on transmission cost. The transmission cost and quality of Omni-based transmission based on SR transmission method in scenario 1, Our progressive approach shows the transmission cost and quality based on the method of this application in scenario 2, Block-based increment (64) shows the transmission cost and quality when 64 blocks are used as local increments, Block-based increment (25) shows the transmission cost and quality when 25 blocks are used as local increments, and Block-based increment (9) shows the transmission cost and quality when 9 blocks are used as local increments. The transmission cost is calculated as the percentage of information increment to the original image, and the transmission quality is represented by SSIM.

[0121] Specifically, the 0817th low-resolution image of the DIV2K dataset is taken as an example to demonstrate the advantages of the method of the present application in local information increment. The size of the region of interest is set to 9, 25, and 64 blocks. The experimental results show that the block-based local information increment effectively reduces the amount of data required to be transmitted, because when transmitting additional information, only the data related to each block is transmitted, not the entire image. When the number of regional blocks increases from 9 to 64, the change in transmission cost tends to be stable, which indicates that for this image (whose total number of blocks is 3060), the increase in blocks does not significantly change the overall transmission cost.

[0122] 3. Quality of super-resolution images:

[0123] As shown in Table 1: The effect of information increment on PSNR / SSIM improvement, as the downsampling information increment increases (i.e., α1-α7), the average values ​​of PSNR / SSIM of the five benchmark data sets (the average value of PSNR on the left and the average value of SSIM on the right) gradually increase. This shows that the method of this application is consistent with expectations and can effectively improve the quality of image reconstruction. In this experiment, the number of pixels contained in each information increment block (i.e., α1-α7) are 17, 19, 21, 23, 25, 27, and 29, respectively.

[0124] Table 1:

[0125]

[0126] As shown in Table 2: The effects of different block sizes on PSNR / SSIM and the number of parameters of the downsampling module. Among them, Params (MB) represents the size of the module parameters (in megabytes). Specifically, although the image quality obtained by the three block sizes is similar, the smaller block size significantly reduces the number of parameters of the downsampling module. However, the smaller block size imposes more restrictions on the progressive performance, resulting in lower progressiveness. Therefore, the choice of block size should be based on the performance requirements and resource trade-offs in the target application scenario.

[0127]

[0128] like Figure 7 : The effect of gradual addition of incremental information on the visual effect of super-resolution images is shown in the schematic diagram, wherein represents a low-resolution image of a picture, and the super-resolution visual effect obtained after the incremental information is gradually added. It is shown that in the DIV2K dataset, the 0817th low-resolution image is used to demonstrate the visual effect of the method of the present application by gradually adding low-resolution information (that is, incremental information). The original Omni surpasses the traditional bicubic interpolation method (bicubic). As the low-resolution information increment increases from α1 to α7, the visual effect of the super-resolution image gradually improves and becomes clearer. This information addition process is flexible and can be terminated early when certain conditions are met, thereby saving transmission costs.

[0129] like Figure 8 : The schematic diagram of the effect of block-based local information increment on super-resolution images is shown in the figure, where: Figure 8 (a) represents the 0831st low-resolution image in DIV2k. Figure 8 (b) Super-resolution image showing the patch-based local information increment.

[0130] Specifically, the red box is the area of ​​interest to the user, and the goal is to clearly identify the numbers in it; the green box is the area of ​​no interest to the user, and no details need to be enhanced. Under the block-based information increment strategy, only the red box area is transmitted. Figure 8 As can be seen from the super-resolution image in (b), the number "67" in the red frame is clearly displayed, while other areas (including the green frame) remain blurred. This verifies that the block-based information increment method can effectively reduce unnecessary transmission costs while meeting local visual quality requirements.

[0131] 4. Ablation experiment:

[0132] As shown in Table 3: Performance ablation experiment results of convolutional layers in image super-resolution framework, the experiment uses the DIV2K dataset. The results show that the introduction of convolutional layers provides a slight performance improvement in PSNR and SSIM indicators. This is because the convolutional layer performs connection operations at the feature level (rather than the pixel level), thereby better learning the key feature information that helps image super-resolution. This experiment verifies the value of convolutional layers in optimizing framework performance.

[0133] Table 3:

[0134]

[0135] like Fig. 9 : The impact of unified and separate training strategies on super-resolution performance is shown in Figure 2. One strategy is to train a unified super-resolution (SR) module for the entire framework (such as Fig. 9 The other is to train a dedicated SR module for each information increment (such as Fig. 9 , as shown in the blue data bar of the bar graph). It can be observed from the figure that the PSNR (peak signal-to-noise ratio) values ​​of the unified training strategy and the separate training strategy are almost the same. This shows that in this framework, the unified training strategy can achieve similar performance without training the SR module separately for each information increment.

[0136] The experimental results verify the progressiveness of the method of this application. The block-based strategy allows information increment only for the area of ​​interest in the image, which greatly improves the practical feasibility of the method and further reduces the bandwidth requirements.

[0137] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0138] In one embodiment, a low-resolution increment-based image super-resolution system is provided, comprising:

[0139] An initial downsampling module is used to perform initial downsampling on the original image to obtain initial downsampling information;

[0140] The initial upsampling module and the super-resolution module are used to perform upsampling and super-resolution processing according to the initial downsampling information to obtain a restored image; downsampling refers to the process of reducing the resolution of a high-resolution image, and upsampling refers to the process of increasing the resolution of a low-resolution image;

[0141] An additional downsampling module is used to perform additional downsampling based on an on-demand mechanism in the current incremental reconstruction round to obtain information increment;

[0142] An additional upsampling module and a super-resolution module are used to perform upsampling and super-resolution processing according to the information increment and the initial downsampling information to obtain a reconstructed image;

[0143] The additional downsampling module, the additional upsampling module and the super-resolution module are also used to perform additional downsampling based on an on-demand mechanism to obtain the next information increment and enter the reconstruction of the next incremental reconstruction round until the final reconstructed image meets the requirements and the reconstruction is stopped.

[0144] For the specific definition of the image super-resolution system based on low-resolution increments, please refer to the definition of the image super-resolution method based on low-resolution increments above, which will not be repeated here. Each module in the above-mentioned image super-resolution system based on low-resolution increments can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0145] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0146] Perform initial downsampling on the original image to obtain initial downsampling information, perform upsampling and super-resolution processing based on the initial downsampling information to obtain a restored image; downsampling refers to the process of reducing the resolution of a high-resolution image, and upsampling refers to the process of increasing the resolution of a low-resolution image;

[0147] In the current incremental reconstruction round, additional downsampling is performed based on an on-demand mechanism to obtain information increment, and upsampling and super-resolution processing are performed based on the information increment and the initial downsampling information to obtain a reconstructed image;

[0148] Additional downsampling is performed based on an on-demand mechanism to obtain the next information increment, and the reconstruction of the next incremental reconstruction round is started until the final reconstructed image meets the requirements, at which time the reconstruction is stopped.

[0149] It can be understood that in addition to the memory and processor involved above, the above-mentioned computer device also includes other software and hardware components not listed in this specification. The specific components can be determined according to the model of the specific computer device in different application scenarios, and this specification will not list them one by one in detail.

[0150] In one embodiment, when the processor executes the computer program, the added steps or sub-steps in each embodiment of the above-mentioned image super-resolution method based on low-resolution increment can also be implemented.

[0151] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image super-resolution method based on low-resolution increments, characterized in that: include: Performing initial downsampling on the original image to obtain initial downsampling information, and performing upsampling and super-resolution processing according to the initial downsampling information to obtain a restored image; Downsampling refers to the process of reducing the resolution of a high-resolution image, and upsampling refers to the process of increasing the resolution of a low-resolution image. In the current incremental reconstruction round, additional downsampling is performed based on an on-demand mechanism to obtain information increment, and upsampling and super-resolution processing are performed according to the information increment and the initial downsampling information to obtain a reconstructed image; Additional downsampling is performed based on an on-demand mechanism to obtain the next information increment, and the reconstruction of the next incremental reconstruction round is started until the last reconstructed image meets the requirements, at which time the reconstruction is stopped.

2. The image super-resolution method based on low-resolution increment according to claim 1, characterized in that: The initial downsampling process is: Among them, p k represents the kth downsampled low-resolution pixel, denotes a convolution operation, φ(k) denotes the kth channel of the convolution kernel φ, Q(·) denotes a quantization module, and X denotes an H×W image.

3. The image super-resolution method based on low-resolution increment according to claim 1 or 2, characterized in that: The on-demand mechanism includes: Judging whether the reconstructed image meets the requirements according to the visual effect of the reconstructed image or the no-reference index; When it is determined that the requirements are not met, the image area to be supplemented and its corresponding sampling increment are determined, and a request is sent to the sender; The sender performs additional downsampling according to the request using a block-based downsampling strategy to obtain the information increment.

4. The image super-resolution method based on low-resolution increment according to claim 3, characterized in that: The initial downsampling information is: Wherein, α0 represents the initial downsampling information, J represents that the original image is divided into J blocks, represents the number of low-resolution pixels contained in the initial downsampling information of the j-th block of the original image, represents the kth down-sampled low-resolution pixel of the jth block of the original image, and B j Denoted as the j-th block of the original image.

5. The image super-resolution method based on low-resolution increment according to claim 4, characterized in that: The information increment is: Among them, α i represents the information increment of the i-th additional downsampling, wherein, represents the total number of low-resolution pixels accumulated in the i-1th round of the jth block of the original image, Represents the number of low-resolution pixels contained in the i-th round of the sampling increment of the j-th block of the original image, wherein the convolution kernel is initially designed to have n channels, and the sampling increment is obtained by expanding m channels.

6. The image super-resolution method based on low-resolution increment according to claim 5, characterized in that: The upsampling process comprises: Wherein, LR0 represents the visual form information obtained in the initial downsampling; R(·) represents the rearrangement operation; LR i represents the visual form information obtained in the i-th incremental reconstruction round; C(·,·) represents a connection operation, and conv(·) represents a convolutional layer; An upsampling and a second rearrangement operation is performed according to the visual form information.

7. The image super-resolution method based on low-resolution increment according to claim 1, characterized in that: Before the step of initially downsampling the original image, the method further includes: The image super-resolution model is trained, and the training process includes: Jointly training an initial downsampling module, an initial upsampling module, and a super-resolution module, optimizing through a loss function until convergence and then fixing the parameters of the initial downsampling module; Continue to train the additional downsampling module, the additional upsampling module and the super-resolution module, and optimize the loss function until convergence to obtain the trained image super-resolution model.

8. The image super-resolution method based on low-resolution increment according to claim 7, characterized in that: The loss function is: Among them, mse represents mean square error, l1 represents standard l1 norm loss, LR i represents the visual form information obtained in the i-th incremental reconstruction round, and LR represents LR i The real reference, SR i represents the reconstructed image obtained in the i-th incremental reconstruction round, and HR represents the original image.

9. An image super-resolution system based on low-resolution increments, characterized in that: include: An initial downsampling module is used to perform initial downsampling on the original image to obtain initial downsampling information; An initial upsampling module and a super-resolution module, used to perform upsampling and super-resolution processing according to the initial downsampling information to obtain a restored image; downsampling refers to the processing of reducing the resolution of a high-resolution image, and upsampling refers to the processing of increasing the resolution of a low-resolution image; An additional downsampling module is used to perform additional downsampling based on an on-demand mechanism in the current incremental reconstruction round to obtain information increment; An additional upsampling module and the super-resolution module are used to perform upsampling and super-resolution processing according to the information increment and the initial downsampling information to obtain a reconstructed image; The additional downsampling module, the additional upsampling module and the super-resolution module are also used to perform additional downsampling based on an on-demand mechanism to obtain the next information increment and enter the reconstruction of the next incremental reconstruction round until the last reconstructed image meets the requirements and the reconstruction is stopped.

10. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the processor executes the computer program, the steps of the image super-resolution method based on low-resolution increments described in any one of claims 1 to 8 are implemented.

Images