Low-resolution increment-based image super-resolution methods, systems, and devices

Abstract

The application relates to an image super-resolution method, system and device based on low-resolution increments, which comprises initial down-sampling and multiple rounds of incremental reconstruction. Each round of incremental reconstruction relies on additional down-sampling to obtain an information increment. Based on the information increment, up-sampling and super-resolution processing are performed until a required reconstructed image is obtained. Through multiple rounds of on-demand additional down-sampling, the region of interest of a user (which can be an entire image or one or more local image blocks) is focused, incremental optimization is realized, transmission of redundant data is effectively reduced, unnecessary resolution overhead is avoided, dependence on the size of a low-resolution image is reduced, data transmission and processing are ensured to be more concise, system efficiency is improved, and bandwidth is saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image super-resolution method, system and device based on low-resolution increment. BACKGROUND

[0002] The continuous improvement of camera resolution brings people a better visual experience. However, even if a high-performance compression algorithm is used, transmitting a large number of large-size high-resolution images on the Internet will still bring huge pressure to the core network and access network. In fact, in many scenarios, due to the limited screen size of display devices (such as smart phones), the original resolution image is not needed. The image super-resolution method effectively solves this problem: the sender can down-sample the original image and transmit its low-resolution version to match the size of the target display device; if necessary, the receiver can restore the image to the original resolution size through the image super-resolution (SR) technology.

[0003] There are many well-designed models for image super-resolution (SR), which focus on improving the quality of the super-resolution image. However, the quality of the super-resolution image is heavily dependent on and subject to the super-resolution magnification scale, that is, the closer the size of the low-resolution image to the original image, the higher the quality of the super-resolution image. In other words, when the quality of the super-resolution image cannot meet the user's demand, the quality can be improved by retransmitting a larger size low-resolution image version, but this will result in redundant transmission and bandwidth waste.

[0004] That is, the existing image super-resolution method has the technical problems of dependence on low-resolution image size, redundant transmission and bandwidth waste. SUMMARY

[0005] Therefore, it is necessary to provide an image super-resolution method based on low-resolution increment, an image super-resolution system based on low-resolution increment and a computer device in view of the above technical problems.

[0006] In order to achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0007] On the one hand, an image super-resolution method based on low-resolution increment is provided, comprising:

[0008] initially down-sampling the original image to obtain initial down-sampling information, and performing up-sampling and super-resolution processing according to the initial down-sampling information to obtain a restored image; the down-sampling refers to a processing of reducing the resolution of a high-resolution image, and the up-sampling refers to a processing of improving the resolution of a low-resolution image;

[0009] In the current incremental reconstruction round, additional downsampling is performed based on an on-demand mechanism to obtain an information increment, upsampling and super-resolution processing are performed based on the information increment and the initial downsampled information to obtain a reconstructed image;

[0010] Additional downsampling is performed based on an on-demand mechanism to obtain a next information increment, and reconstruction of the next incremental reconstruction round is entered until the final reconstructed image meets the requirements, and the reconstruction is stopped.

[0011] In one of the embodiments, the image super-resolution method based on low-resolution increments further includes that the process of initial downsampling is:

[0012]

[0013] wherein, p k represents the kth downsampled low-resolution pixel of the jth block of the original image, represents a convolution operation, φ(k) represents the kth channel of the convolution kernel φ, Q(·) represents a quantization module, and X represents an HxW image.

[0014] In one of the embodiments, the image super-resolution method based on low-resolution increments includes that the on-demand mechanism includes:

[0015] According to the visual effect or the no-reference index of the reconstructed image, it is determined whether the reconstructed image meets the requirements;

[0016] When it is determined that the requirements are not met, the image region to be supplemented and the corresponding sampling increment are determined, and a request is sent to the sender;

[0017] The sender performs additional downsampling based on a block-based downsampling strategy according to the request to obtain an information increment.

[0018] In one of the embodiments, the image super-resolution method based on low-resolution increments includes that the initial downsampled information is:

[0019]

[0020] wherein, α0 represents the initial downsampled information, J represents that the original image is divided into J blocks, represents the number of low-resolution pixels contained in the initial downsampled information of the jth block of the original image, represents the kth downsampled low-resolution pixel of the jth block of the original image, and B j represents the jth block of the original image.

[0021] In one of the embodiments, the image super-resolution method based on low-resolution increments includes that the information increment is:

[0022]

[0023] wherein, a i denotes the information increment of the i-th additional down-sampling, wherein, denotes the total number of low-resolution pixels accumulated in the i-1-th round of the j-th block of the original image, denotes the number of low-resolution pixels contained in the i-th round of the sampling increment of the j-th block of the original image, wherein the convolution kernel is initially designed with n channels, and the sampling increment is obtained by extending m channels.

[0024] In one embodiment, the low-resolution increment-based image super-resolution method, the up-sampling process comprises:

[0025]

[0026] wherein, LR0denotes the visual form information obtained in the initial down-sampling; R(·) denotes the rearrangement operation; LR i denotes the visual form information obtained in the i-th increment reconstruction round; C(·,·) denotes the concatenation operation, and conv(·) denotes the convolution layer; the up-sampling and the second rearrangement operation are performed according to the visual form information.

[0027] In one embodiment, the low-resolution increment-based image super-resolution method, before the step of performing the initial down-sampling on the original image, further comprises:

[0028] training the image super-resolution model, the training process comprising:

[0029] jointly training the initial down-sampling module, the initial up-sampling module and the super-resolution module, optimizing through the loss function until convergence, and then fixing the parameters of the initial down-sampling module;

[0030] continuing to train the additional down-sampling module, the additional up-sampling module and the super-resolution module, optimizing through the loss function until convergence, and obtaining the trained image super-resolution model.

[0031] In one embodiment, the low-resolution increment-based image super-resolution method, the loss function is:

[0032]

[0033] wherein, mse denotes the mean square error, l1denotes the standard l1norm loss, LR i denotes the visual form information obtained in the i-th increment reconstruction round, and LR i denotes the true reference of LR i denotes the reconstructed image obtained in the i-th increment reconstruction round, and HR denotes the original image.

[0034] On the other hand, a low-resolution increment-based image super-resolution system is also provided, comprising:

[0035] The initial downsampling module is used to perform initial downsampling on the original image to obtain initial downsampling information;

[0036] The initial upsampling module and super-resolution module are used to perform upsampling and super-resolution processing based on the initial downsampling information to obtain a restored image; downsampling refers to the resolution reduction process of a high-resolution image, and upsampling refers to the resolution increase process of a low-resolution image;

[0037] An additional downsampling module is used to perform additional downsampling based on an on-demand mechanism in the current incremental reconstruction round to obtain information increments;

[0038] An additional upsampling module and a super-resolution module are used to perform upsampling and super-resolution processing according to the information increment and the initial downsampling information to obtain a reconstructed image;

[0039] The additional downsampling module, the additional upsampling module and the super-resolution module are also used to perform additional downsampling based on an on-demand mechanism to obtain the next information increment and enter the reconstruction of the next incremental reconstruction round until the final reconstructed image meets the requirements and the reconstruction is stopped.

[0040] On the other hand, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned image super-resolution method based on low-resolution increments when executing the computer program.

[0041] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0042] The aforementioned low-resolution incremental image super-resolution method, system, and device employ initial downsampling and multiple rounds of incremental reconstruction. Each round of incremental reconstruction relies on additional downsampling to obtain information increments. Based on these information increments, upsampling and super-resolution processing are then performed until a reconstructed image that meets the requirements is obtained. Through multiple rounds of on-demand additional downsampling, the system focuses on the user's area of ​​interest (either the entire image or one or more local image blocks), achieving incremental optimization. This effectively reduces the transmission of redundant data, avoids unnecessary resolution overhead, and reduces reliance on low-resolution image size. This ensures more streamlined data transmission and processing, improves system efficiency, and conserves bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0044] Figure 1 Flowchart of an embodiment of low-resolution increment-based image super-resolution method;

[0045] Figure 2 Flowchart of an embodiment of low-resolution increment-based image super-resolution framework;

[0046] Figure 3 Block-based convolution structure in an embodiment;

[0047] Figure 4 SSIM-based super-resolution image transmission cost comparison in an embodiment, wherein (a) represents the comparison result obtained using dataset Set5, (b) represents the comparison result obtained using dataset Set14, (c) represents the comparison result obtained using dataset BSDS100, (d) represents the comparison result obtained using dataset Urban100, and (e) represents the comparison result obtained using dataset DIV2K;

[0048] Figure 5 PSNR-based super-resolution image transmission cost comparison in an embodiment, wherein (a) represents the comparison result obtained using dataset Set5, (b) represents the comparison result obtained using dataset Set14, (c) represents the comparison result obtained using dataset BSDS100, (d) represents the comparison result obtained using dataset Urban100, and (e) represents the comparison result obtained using dataset DIV2K;

[0049] Figure 6 Block-based local information increment impact on transmission cost in an embodiment;

[0050] Figure 7 Increment information step-by-step addition impact on super-resolution image visual effect in an embodiment;

[0051] Figure 8 Block-based local information increment impact on super-resolution image in an embodiment, wherein, Figure 8 (a) represents the 0831th low-resolution image in DIV2k, Figure 8 (b) represents the block-based local information increment-based super-resolution image;

[0052] Figure 9 For one embodiment, the impact of unified and separate training strategies on super-resolution performance is illustrated. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0055] It should be noted that reference to "an embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same embodiment, or to a single alternative embodiment.

[0056] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0057] In one embodiment, as shown in Figure 1 The embodiment of the present application provides a low-resolution increment-based image super-resolution method, which includes the following processing steps S12-S16.

[0058] S12, performing initial down-sampling on the original image to obtain initial down-sampling information, and performing up-sampling and super-resolution processing according to the initial down-sampling information to obtain a restored image; down-sampling refers to a resolution reduction processing of a high-resolution image, and up-sampling refers to a resolution improvement processing of a low-resolution image.

[0059] It can be understood that the initial down-sampling is to reduce the resolution of the high-resolution original image to extract the initial down-sampling information, and then use the initial down-sampling information to improve the resolution of the low-resolution image through up-sampling and super-resolution processing to generate a restored image.

[0060] Specifically, the downsampling can be implemented by mean pooling, bilinear interpolation, or convolution operation. The upsampling can use deconvolution, up-sampling layer, or sub-pixel convolution operation to increase the resolution of the downsampling information to the size of the target resolution. Through a super-resolution network model, such as SRCNN (Super-Resolution Convolutional Neural Network), etc., the up-sampled image is further optimized to generate a restored image. The network can also combine adversarial learning, perceptual loss, or self-supervised learning to improve the clarity and authenticity of the image.

[0061] S14, in the current incremental reconstruction round, additional downsampling is performed based on the on-demand mechanism to obtain an information increment, and upsampling and super-resolution processing are performed based on the information increment and the initial downsampling information to obtain a reconstructed image.

[0062] It can be understood that in the process of the current incremental reconstruction, the high-resolution original image is subjected to on-demand additional downsampling to generate low-resolution reference data, which is referred to as an information increment. The information increment and the initial downsampling information are subjected to upsampling, and finally a new reconstructed image is generated by further optimization using a super-resolution algorithm.

[0063] Specifically, the downsampling based on the on-demand mechanism can be performed based on a specific region to obtain additional supplementary information as the information increment. The specific region is a region of interest of the user (which can be the entire image or one or more local image blocks). The information increment and the initial downsampling information are fused, for example, by weighting or splicing, and then subjected to upsampling and super-resolution processing to output a high-quality reconstructed image.

[0064] S16, additional downsampling is performed based on the on-demand mechanism to obtain a next information increment, and the reconstruction of the next incremental reconstruction round is entered until the final reconstructed image meets the requirements, and the reconstruction is stopped.

[0065] It can be understood that the next information increment is obtained, and the reconstruction of the next incremental reconstruction round is entered. Specifically, the incremental reconstruction is performed based on the previously obtained information increment, the initial downsampling information, and the obtained next information increment. The multiple incremental reconstruction rounds are repeated until the final reconstructed image meets the requirements, and the reconstruction is stopped. Specifically, whether the requirements are met can be determined according to the visual effect, or whether the requirements are met can be evaluated by a no-reference index such as PI. The image region of interest of the user is used as the optimization target (which can be the entire image or one or more local image blocks), and it is ensured that the reconstructed result can meet the actual requirements.

[0066] In the above low-resolution increment-based image super-resolution method, through initial down-sampling and multiple rounds of incremental reconstruction, each round of incremental reconstruction relies on additional down-sampling to obtain information increment, and based on the information increment, through up-sampling and super-resolution processing, until a required reconstructed image is obtained. Through multiple rounds of on-demand additional down-sampling, the user's interested area (which can be the entire image or one or more local image blocks) is focused, incremental optimization is achieved, redundant data transmission is effectively reduced, unnecessary resolution overhead is avoided, the dependence on low-resolution image size is reduced, data transmission and processing are ensured to be more concise, system efficiency is improved, and bandwidth is saved.

[0067] In one embodiment, the process of initial down-sampling is:

[0068]

[0069] where p k represents the kth down-sampled low-resolution pixel, represents a convolution operation, φ(k) represents the kth channel of the convolution kernel φ, and Q(·) represents a quantization module, and X represents an HxW image.

[0070] It can be understood that the low-resolution increment-based image super-resolution framework diagram is as shown in Figure 2 , which includes an initial down-sampling module D-S0, an additional down-sampling module D-S i (i≥1, the same as understood below, and will not be repeated), an initial up-sampling module U-S0, an additional up-sampling module U-S i , and a super-resolution module (i.e., an SR module), and the down-sampling module is executed at the sender.

[0071] In the initial down-sampling, the random parameters of the convolution kernel φ can be used to sample the image, and the size of the convolution kernel can be designed as HxW, which matches the height and width of the image to be transmitted. In this application, the block-based convolution is used as a parameter within the h x w block, and the remaining parameters are set to 0. In essence, the block-based convolution still samples the entire image.

[0072] where Q(·) is a quantization module, which is used to convert the original floating-point number (usually a high-precision 32-bit or 16-bit floating-point number) into an 8-bit unsigned integer through quantization processing, because images are usually represented by integers between 0 and 255.

[0073] At the beginning of image X transmission, in order to save bandwidth, the sender first transmits the initial down-sampling information α0of the image. Assuming that α0contains n0low-resolution pixels, α0can be represented as:

[0074]

[0075] If the sender receives a request for m i additional pixels, module D-S i is executed and m i low-resolution pixels are added with information increment a i :

[0076]

[0077] wherein, The process of sending bits in response to the request will be repeated until the user is satisfied. a0and a i represent the initial down-sampling information and the information increment of the i-th additional down-sampling, respectively.

[0078] In one embodiment, the image super-resolution method based on low-resolution increment includes:

[0079] According to the visual effect or the no-reference index of the reconstructed image, it is determined whether the reconstructed image meets the requirements; when it is determined that the requirements are not met, the image area to be supplemented (which can be the entire image or one or more local image blocks) and the corresponding sampling increment are determined, and a request is sent to the sender; the sender performs additional down-sampling based on the block-based down-sampling strategy according to the request to obtain the information increment.

[0080] It can be understood that whether the reconstructed image meets the quality requirements can be determined by the visual effect of the reconstructed image, or by the no-reference index; if not, the image area to be supplemented and the sampling increment are determined, and a request is sent to the sender. The sender performs additional down-sampling based on the block-based down-sampling strategy to generate the information increment for supplementing the reconstruction. In a specific implementation, the system uses an iterative method to gradually optimize the reconstruction effect for the area of interest (which can be the entire image or one or more local image blocks), thereby realizing accurate data acquisition. This method reduces unnecessary data transmission, optimizes data processing efficiency, ensures the quality of the reconstructed image, and reduces transmission costs. In addition, the no-reference index does not need to rely on the original image, and can directly evaluate the quality from the reconstructed image, which is suitable for scenarios where the original data is unknown, thereby improving the universality and real-time performance of the evaluation.

[0081] In one embodiment, the block-based down-sampling strategy, the initial down-sampling information is:

[0082]

[0083] wherein, a0represents the initial down-sampling information, J represents that the original image is divided into J blocks, represents the number of low-resolution pixels contained in the initial down-sampling information of the j-th block of the original image, denotes the k-th down-sampled low-resolution pixel of the j-th block of the original image, and B j denotes the j-th block of the original image.

[0084] It can be understood that the block-based convolution structure is as shown in Figure 3 As shown in the figure, the size of the convolution kernel on the k-th channel is designed as h x w, the block size of the original image is H x W, and the original image is divided into J blocks, where J = rs, r = H / h, and s = W / w.

[0085] The information increment is:

[0086]

[0087] wherein, α i denotes the information increment of the i-th additional down-sampling, wherein, denotes the total number of low-resolution pixels accumulated in the i-1-th round of the j-th block of the original image, denotes the number of low-resolution pixels contained in the i-th round of the sampling increment of the j-th block of the original image, wherein the convolution kernel is initially designed with n channels, and the sampling increment is obtained by expanding m channels.

[0088] wherein, the block-based convolution can be regarded as retaining the parameters within the h x w block, and setting the remaining parameters to 0. Essentially, the block-based convolution still samples the entire image. However, by setting multiple parameters to 0, the block-based strategy avoids the convolution operation with an excessively large number of parameters, making the down-sampling process feasible.

[0089] The above low-resolution increment-based image super-resolution method uses convolution and its random parameters to generate down-sampled low-resolution information, wherein the n channels of the convolution kernel correspond to n times of down-sampling. Since additional m low-resolution pixels are required, the convolution kernel is further expanded by m channels, providing the ability to add low-resolution information increments at any scale. The information increments of multiple iterations are gradually integrated by a carefully designed up-sampling module, and the super-resolution processing is performed by the SR module. Through block division, integration can be performed block by block to achieve on-demand super-resolution.

[0090] In one embodiment, the up-sampling process comprises:

[0091]

[0092] wherein, LR0 denotes the visual form information obtained in the initial down-sampling; R(·) denotes a rearrangement operation; LR irepresents the visual form information obtained in the i-th incremental reconstruction round; C(·,·) represents the concatenation operation, and conv(·) represents the convolution layer; the upsampling and the second rearrangement operation are performed according to the visual form information.

[0093] It can be understood that, in combination with Figure 2 As shown in the figure, in the upsampling process, two rearrangement operations, one concatenation operation and one upsampling operation are included, in the rearrangement operation, a0 is rearranged into its visual form LR0. If there is an additional information increment a1 available, the features of LR0 are concatenated with the new available information increment a1 through the convolution layer conv(·), and the result is rearranged to generate LR1. This process of concatenation and rearrangement can be repeatedly performed until the user's requirements are met. Among them, LR i is the corresponding visual version after the i-th increment.

[0094] Specifically, the upsampling operation is responsible for upsampling the small size low resolution version LR i of the image X to a suitable scale. Since the existing super resolution (SR) module only supports fixed ratio (such as integer multiple) upsampling, and the method of the present application supports upsampling and downsampling of any ratio, therefore, before performing super resolution processing, the LR i can be first adjusted to an adaptive scale through the upsampling module, and then input into the SR module.

[0095] Upsampling is the inverse process of downsampling, and is also achieved by using a convolution structure. Let N = H × W. Then the size of the convolution kernel Φ(k) of the i-th upsampling is n i , and the number of channels is N. The output of N times of convolution operation is:

[0096]

[0097] Among them the N pixels are rearranged to the size of H × W, which is used as the input of the SR module. After super resolution processing by the SR module, the output SR0 and SR i are obtained, where SR0 represents the restored image, and SR i represents the reconstructed image of the i-th incremental reconstruction round.

[0098] For block-based upsampling, the convolution layer conv(·) and the concatenation operation C(·,·) are also based on image blocks, and the second rearrangement operation R(·) further arranges a single block into a complete image, which is then input into the SR module. Through the processing of the SR module, block artifacts can be improved.

[0099] In one embodiment, before the step of initially downsampling the original image, further comprising: training an image super resolution model, the training process comprising:

[0100] The initial down-sampling module, the initial up-sampling module and the super-resolution module are jointly trained, and the parameters of the initial down-sampling module are fixed after convergence through loss function optimization. The additional down-sampling module, the additional up-sampling module and the super-resolution module are continuously trained, and the trained image super-resolution model is obtained after convergence through loss function optimization.

[0101] It can be understood that the image super-resolution model is first trained in stages. In the initial stage, the initial down-sampling module, the initial up-sampling module and the super-resolution module are jointly trained, and the loss function is used for optimization until the model converges. In this stage, the parameters of the initial down-sampling module are continuously adjusted to achieve the best down-sampling effect. Subsequently, the parameters of the initial down-sampling module are fixed, and the second stage is entered to continue training the additional down-sampling module, the additional up-sampling module and the super-resolution module for multiple rounds to further optimize the overall model. Through this stage-by-stage training method, the initial down-sampling module can stably provide basic image information, and the additional modules can further improve the quality of the entire image on this basis. Finally, through the optimization of the loss function, the trained image super-resolution model is obtained.

[0102] The above-mentioned image super-resolution method based on low-resolution increment effectively improves the stability and efficiency of model training. In the initial stage, by optimizing the parameters of the initial down-sampling module, the effective preservation of image information in the down-sampling process is ensured. Then, on the basis of fixing the parameters of the initial down-sampling module, the remaining modules are further trained to improve the quality of the entire image. More accurate image restoration effect can be achieved, and the generalization ability and application effect of the model are improved.

[0103] In one embodiment, the loss function of the image super-resolution method based on low-resolution increment is:

[0104]

[0105] wherein mse represents the mean square error, l1 represents the standard l1 norm loss, LR i represents the visual form information obtained in the i-th incremental reconstruction round, LR i represents the true reference of LR i represents the reconstructed image obtained in the i-th incremental reconstruction round.

[0106] It can be understood that, in addition to using the loss function described above, a more complex feature-level perceptual loss can also be used to improve performance, so as to not only focus on the pixel difference of the image, but also consider the perceptual quality and high-level features (such as edges, textures, structures, etc.) of the image. Specifically, the feature-level perceptual loss is usually extracted by using a pre-trained deep neural network, for example, a VGG (Visual Geometry Group) network, to extract high-level features of the image. Some layers in the VGG network can be used to extract intermediate features of the image (such as the output of the convolutional layer), and then the difference between the reconstructed image and the original image in these features is calculated, instead of relying only on the pixel difference. Thus, it is closer to the perception of the human visual system, and therefore can provide a more natural and delicate image restoration effect.

[0107] In some embodiments, in order to more intuitively and comprehensively illustrate the above-mentioned low-resolution increment-based image super-resolution method, the following is an application example of the low-resolution increment-based image super-resolution method. It should be noted that the implementation cases given in the present specification are only illustrative, and are not the only limitation of the specific implementation cases of the present application. Those skilled in the art can use the above-mentioned low-resolution increment-based image super-resolution method to reduce the dependence on the size of the low-resolution image and focus on the region of interest of the user to achieve incremental optimization and effectively reduce the transmission of redundant data under the guidance of the illustrative implementation cases provided by the present application.

[0108] The latest transformer-based SR network Omni is used as the SR module. It is worth noting that any other SR network can also be used as the SR module in a similar manner.

[0109] 1. Experimental setup:

[0110] Data sets and indicators. DIV2K and Flickr2K are used as training data sets. DIV2K provides 1000 images with a resolution of up to 2K, while Flickr2K contains 2650 high-quality images collected from the Flickr platform. These data sets support the model to achieve excellent performance in the super-resolution task through rich image details and diversity. In order to speed up the image loading during the training process, each training image is evenly divided into four parts, and each part becomes an independent, relatively small training image.

[0111] For testing purposes, we use five standard benchmark datasets, including Set5, Set14, BSDS100, Urban100, and DIV2K validation set. Set5 and Set14 provide classic low-resolution-high-resolution image pairs, suitable for preliminary verification of model effectiveness; BSDS100 contains more complex natural scene images, testing the performance of the model on diversified content; Urban100 focuses on urban buildings and structural textures, challenging the model's detail restoration ability; DIV2K validation set provides a modern benchmark for high-resolution images, measuring the applicability of the model in high-resolution scenarios. Through these diverse datasets, the performance of the model in different scenarios can be comprehensively evaluated.

[0112] The quality of the super-resolution image in the Y channel of the transformed YCbCr space (luminance and chrominance space) is evaluated using PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity).

[0113] Implementation details: The training images are augmented by random flipping and 90-degree rotation. The ground truth (true reference) of the ×4 low-resolution version is generated from the corresponding original image by bicubic interpolation. The proposed framework is trained using the AdamW optimizer (Weight Decay Adam with Momentum Optimizer) with a batch size of 16 and 300K iterations. The initial learning rate is set to 0.0005, and it is halved every 50K iterations. In each training batch, the original high-resolution image is randomly cropped into small blocks of size 64×64 as input. All experiments are conducted on a computer equipped with an NVIDIA GeForce RTX 4090 GPU, an Intel Core i7-14700KF processor with a frequency of 3.40 GHz, and 64 GB of memory using PyTorch 2.2.0.

[0114] Block size. The downsampling module uses a block-based convolution structure. The block size of the original image and its ×4 low-resolution version is set to 32×32 and 8×8, respectively.

[0115] 2. Transmission cost:

[0116] The transmission cost is estimated using the ratio of the downsampling information increment to the original image. Scenario 1: SR-based transmission method: if the super-resolution image does not meet the requirements, a larger low-resolution version must be retransmitted. Scenario 2: using the method of the present application, additional downsampling is performed based on the on-demand mechanism, specifically using a block-based convolution structure. To ensure fair comparison, both scenarios 1 and 2 use Omni.

[0117] Figure 4Figures for comparison of transmission cost of super-resolution image based on SSIM, where (a) represents the comparison result obtained using dataset Set5, (b) represents the comparison result obtained using dataset Set14, (c) represents the comparison result obtained using dataset BSDS100, (d) represents the comparison result obtained using dataset Urban100, and (e) represents the comparison result obtained using dataset DIV2K. Wherein, Omni-based transmission represents the transmission cost and quality based on SR transmission method under scenario 1, and Our progressive approach represents the transmission cost and quality based on the method of the present application under scenario 2. The transmission cost is calculated as the percentage of information increment over the original image, and the transmission quality is represented by SSIM. Proportion represents the percentage.

[0118] Figure 5 Figures for comparison of transmission cost of super-resolution image based on PSNR, where (a) represents the comparison result obtained using dataset Set5, (b) represents the comparison result obtained using dataset Set14, (c) represents the comparison result obtained using dataset BSDS100, (d) represents the comparison result obtained using dataset Urban100, and (e) represents the comparison result obtained using dataset DIV2K. Wherein, General SR-based transmission also represents the transmission cost based on SR transmission method under scenario 1, and Our progressive SR-based transmission also represents the transmission cost based on the method of the present application under scenario 2. The transmission cost is calculated as the percentage of information increment over the original image, and the transmission quality is represented by PSNR. Data size represents the data size (percentage).

[0119] Figures 4-5 In the above, for each benchmark dataset, the first point from the left in the figure represents the transmission cost a0 of the initial low-resolution version. At this time, the methods of scenario 1 and scenario 2 are the same in terms of transmission cost. However, when the super-resolution image quality corresponding to a0 is insufficient, the method of scenario 1 needs to retransmit a larger low-resolution image with a size of a0+a1, while the method of scenario 2 (the present application) only increases an increment a1. The results show that, under the same super-resolution image quality, the method of the present application has significantly lower transmission cost; in addition, as the image quality gradually increases, the savings in transmission cost of the method of the present application are more obvious.

[0120] Figure 6Fig. 1 shows a schematic diagram of the impact of block-based local information increment on transmission cost, where Omni-based transmission is the transmission cost and quality based on SR transmission method under scenario 1, Our progressive approach is the transmission cost and quality based on the method of the present application under scenario 2, Block-based increment (64) is the transmission cost and quality when using 64 blocks as local increment, Block-based increment (25) is the transmission cost and quality when using 25 blocks as local increment, and Block-based increment (9) is the transmission cost and quality when using 9 blocks as local increment. The transmission cost is calculated as the percentage of information increment over the original image, and the transmission quality is represented by SSIM.

[0121] Specifically, taking the 0817th low-resolution image of the DIV2K dataset as an example, the advantages of the method of the present application in local information increment are demonstrated. The size of the region of interest is set to 9, 25 and 64 blocks. The experimental results show that the block-based local information increment effectively reduces the amount of data required for transmission, because when transmitting additional information, only the data related to each block is transmitted, rather than the entire image. When the number of region blocks increases from 9 to 64, the change in transmission cost tends to be stable, which indicates that for this image (with a total of 3060 blocks), the increase in blocks does not significantly change the overall transmission cost.

[0122] 3. Quality of the super-resolution image:

[0123] As shown in Table 1: Impact of information increment on the improvement of PSNR / SSIM, with the increase of down-sampling information increment (i.e. a1-a7), the average value of PSNR / SSIM of the 5 benchmark datasets (the left side is the average value of PSNR, and the right side is the average value of SSIM) gradually increases. This shows that the method of the present application is consistent with the expectation, and can effectively improve the image reconstruction quality. In this experiment, the number of pixels contained in each information increment block (i.e. a1-a7) is 17, 19, 21, 23, 25, 27 and 29, respectively.

[0124] Table 1:

[0125]

[0126] As shown in Table 2: the influence of different block sizes on PSNR / SSIM and the parameter quantity of the downsampling module. Among them, Params(MB) represents the size of the module parameter quantity (in megabytes). Specifically, although the image quality obtained by the three block sizes is similar, the smaller block size significantly reduces the parameter quantity of the downsampling module. However, smaller block size imposes more restrictions on progressive performance, resulting in lower progressiveness. Therefore, the selection of block size should be based on the performance requirements and resource trade-offs in the target application scenario.

[0127]

[0128] As Figure 7 : The influence of incremental information on the visual effect of super-resolution images is shown in the schematic diagram, where the low-resolution image of a picture is represented by Lr, and the super-resolution visual effect obtained after the incremental information is added is represented by Om. The visual effect of the method of the present application is demonstrated by gradually adding low-resolution information (i.e., incremental information) using the 0817th low-resolution image in the DIV2K dataset. The original Omni outperforms the traditional bicubic interpolation method. As the low-resolution information increment increases from a1 to a7, the visual effect of the super-resolution image gradually improves and becomes clearer. This information addition process is flexible and can be terminated early under certain conditions, thereby saving transmission costs.

[0129] As Figure 8 : The influence of block-based local information increment on super-resolution images is shown in the schematic diagram, where Figure 8 (a) represents the 0831th low-resolution image in DIV2k, Figure 8 (b) represents the super-resolution image based on block-based local information increment.

[0130] Specifically, the red box is the region of interest to the user, and the goal is to clearly identify the numbers in it; while the green box is the region not of interest to the user, and there is no need to enhance the details. Under the block-based information increment strategy, only the information increment in the red box region is transmitted. From Figure 8 As can be seen from the super-resolution image in (b), the number "67" in the red box is clearly displayed, while the other regions (including the green box part) remain blurred. It is verified that the block-based information increment method can effectively reduce unnecessary transmission costs while meeting the local visual quality requirements.

[0131] 4. Ablation experiment:

[0132] As shown in Table 3: Convolutional layer performance ablation experiment results in image super-resolution framework, the experiment uses the DIV2K dataset, and the results show that the introduction of the convolutional layer provides a slight performance improvement in the PSNR and SSIM indicators. Because the convolutional layer performs connection operations at the feature level (rather than the pixel level), it better learns key feature information that helps image super-resolution. This experiment verifies the value of the convolutional layer in optimizing the performance of the framework.

[0133] Table 3:

[0134]

[0135] As Figure 9 : The effect of unified and separate training strategies on super-resolution performance is shown in the figure. One strategy is to train a unified super-resolution (SR) module for the entire framework (as shown in the red data bar of the Unified SR model in Figure 9 ), and the other is to train a dedicated SR module for each information increment (as shown in the blue data bar of the separately-trained SR model in Figure 9 ). As can be observed from the figure, the unified training strategy and the separate training strategy are almost identical in terms of PSNR (peak signal-to-noise ratio) value. This indicates that in this framework, the unified training strategy can achieve similar performance without the need to train an SR module for each information increment.

[0136] The experimental results verify the progressiveness of the method of the present application. The block-based strategy allows information increments only for the regions of interest of the image, greatly improving the practical feasibility of the method and further reducing the bandwidth requirement.

[0137] It should be understood that although the steps in the flowchart of Figure 1 are shown in sequence according to the arrows, these steps are not necessarily executed in sequence according to the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least some of the steps in may include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least some of the other steps or sub-steps or stages of other steps.

[0138] In one embodiment, a low-resolution increment-based image super-resolution system is provided, comprising:

[0139] The initial downsampling module is used to perform initial downsampling on the original image to obtain initial downsampling information;

[0140] The initial upsampling module and super-resolution module are used to perform upsampling and super-resolution processing based on the initial downsampling information to obtain a restored image; downsampling refers to the resolution reduction process of a high-resolution image, and upsampling refers to the resolution increase process of a low-resolution image;

[0141] An additional downsampling module is used to perform additional downsampling based on an on-demand mechanism in the current incremental reconstruction round to obtain information increments;

[0142] An additional upsampling module and a super-resolution module are used to perform upsampling and super-resolution processing according to the information increment and the initial downsampling information to obtain a reconstructed image;

[0143] The additional downsampling module, the additional upsampling module and the super-resolution module are also used to perform additional downsampling based on an on-demand mechanism to obtain the next information increment and enter the reconstruction of the next incremental reconstruction round until the final reconstructed image meets the requirements and the reconstruction is stopped.

[0144] Regarding the specific definition of the image super-resolution system based on low-resolution increments, please refer to the definition of the image super-resolution method based on low-resolution increments above, and will not be repeated here. Each module in the above-mentioned image super-resolution system based on low-resolution increments can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0145] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0146] Perform initial downsampling on the original image to obtain initial downsampling information, and then perform upsampling and super-resolution processing based on the initial downsampling information to obtain a restored image. Downsampling refers to the process of reducing the resolution of a high-resolution image, and upsampling refers to the process of increasing the resolution of a low-resolution image.

[0147] In the current incremental reconstruction round, additional downsampling is performed based on the on-demand mechanism to obtain information increment, and upsampling and super-resolution processing are performed based on the information increment and the initial downsampling information to obtain the reconstructed image;

[0148] Additional downsampling is performed based on the on-demand mechanism to obtain the next information increment, and the reconstruction of the next incremental reconstruction round is started until the final reconstructed image meets the requirements, at which point the reconstruction is stopped.

[0149] It can be understood that the above computer device includes other hardware and software components not listed in the specification in addition to the above-mentioned memory and processor, and the specific computer device model can be determined according to different application scenarios, and the specification will not be described in detail.

[0150] In one embodiment, the processor also implements the steps or sub-steps added in each embodiment of the above-mentioned low-resolution incremental-based image super-resolution method when executing the computer program.

[0151] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist Contradiction, it should be considered as the scope of the present application.

[0152] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An image super-resolution method based on low-resolution increments, characterized in that: include: Perform initial downsampling on the original image to obtain initial downsampling information, and perform upsampling and super-resolution processing based on the initial downsampling information to obtain a restored image; downsampling refers to the process of reducing the resolution of a high-resolution image, and upsampling refers to the process of increasing the resolution of a low-resolution image; the initial downsampling information is: Wherein, α0 represents the initial downsampling information, J represents that the original image is divided into J blocks, represents the number of low-resolution pixels contained in the initial downsampling information of the j-th block of the original image, represents the kth down-sampled low-resolution pixel of the jth block of the original image, and B j is represented as the jth block of the original image; the upsampling process includes: Where LR0 represents the visual form information obtained in the initial downsampling; R(·) represents the rearrangement operation; LR i represents the visual form information obtained in the i-th incremental reconstruction round; C(·,·) represents a concatenation operation, and conv(·) represents a convolutional layer; upsampling and a second rearrangement operation are performed according to the visual form information; In a current incremental reconstruction round, additional downsampling is performed based on an on-demand mechanism to obtain information increments, and upsampling and super-resolution processing are performed based on the information increments and the initial downsampling information to obtain a reconstructed image. The on-demand mechanism includes: judging whether the reconstructed image meets requirements based on a visual effect of the reconstructed image or a no-reference indicator; when it is judged that the requirements are not met, determining an image area to be supplemented and its corresponding sampling increment, and sending a request to a sender; the sender performs additional downsampling based on the request using a block-based downsampling strategy to obtain the information increment; the information increment is: Among them, α i represents the information increment of the i-th additional downsampling, wherein, represents the total number of low-resolution pixels accumulated in the i-1th round of the jth block of the original image, represents the number of low-resolution pixels contained in the i-th sampling increment of the j-th block of the original image, wherein the convolution kernel is initially designed to have n channels, and the sampling increment is obtained by expanding m channels; Additional downsampling is performed based on an on-demand mechanism to obtain the next information increment, and the reconstruction of the next incremental reconstruction round is started until the final reconstructed image meets the requirements, at which time the reconstruction is stopped.

2. The image super-resolution method based on low-resolution increment according to claim 1, characterized in that: The initial downsampling process is: Among them, p k represents the kth downsampled low-resolution pixel, represents the convolution operation, φ(k) represents the kth channel of the convolution kernel φ, Q(·) represents the quantization module, and X represents an H×W image.

3. The image super-resolution method based on low-resolution increment according to claim 1, characterized in that: Before the step of performing initial downsampling on the original image, the method further includes: The image super-resolution model is trained. The training process includes: Jointly training an initial downsampling module, an initial upsampling module, and a super-resolution module, optimizing the initial downsampling module through a loss function until convergence, and then fixing the parameters of the initial downsampling module; Continue training the additional downsampling module, the additional upsampling module, and the super-resolution module, and optimize the loss function until convergence to obtain the trained image super-resolution model.

4. The image super-resolution method based on low-resolution increments according to claim 3, characterized in that: The loss function is: Among them, mse represents mean square error, l1 represents standard l1 norm loss, LR i represents the visual form information obtained in the i-th incremental reconstruction round, and LR represents LR i The real reference, SR i represents the reconstructed image obtained in the i-th incremental reconstruction round, and HR represents the original image.

5. An image super-resolution system based on low-resolution increments, characterized in that: include: The initial downsampling module is used to perform initial downsampling on the original image to obtain initial downsampling information; The initial downsampling information is: Wherein, α0 represents the initial downsampling information, J represents that the original image is divided into J blocks, represents the number of low-resolution pixels contained in the initial downsampling information of the j-th block of the original image, represents the kth down-sampled low-resolution pixel of the jth block of the original image, and B j Denoted as the j-th block of the original image; The initial upsampling module and the super-resolution module are used to perform upsampling and super-resolution processing based on the initial downsampling information to obtain a restored image; downsampling refers to the resolution reduction processing of a high-resolution image, and upsampling refers to the resolution improvement processing of a low-resolution image; the upsampling process includes: Where LR0 represents the visual form information obtained in the initial downsampling; R(·) represents the rearrangement operation; LR i represents the visual form information obtained in the i-th incremental reconstruction round; C(·,·) represents a concatenation operation, and conv(·) represents a convolutional layer; upsampling and a second rearrangement operation are performed according to the visual form information; The additional downsampling module is configured to perform additional downsampling based on an on-demand mechanism in a current incremental reconstruction round to obtain information increments; the additional upsampling module and the super-resolution module are configured to perform upsampling and super-resolution processing based on the information increments and the initial downsampling information to obtain a reconstructed image; the on-demand mechanism includes: judging whether the reconstructed image meets requirements based on the visual effect of the reconstructed image or a reference index; when it is judged that the requirements are not met, determining the image area to be supplemented and the corresponding sampling increment, and sending a request to the sender; the sender performs additional downsampling based on the request using a block-based downsampling strategy to obtain the information increment; the information increment is: Among them, α i represents the information increment of the i-th additional downsampling, wherein, represents the total number of low-resolution pixels accumulated in the i-1th round of the jth block of the original image, represents the number of low-resolution pixels contained in the i-th sampling increment of the j-th block of the original image, wherein the convolution kernel is initially designed to have n channels, and the sampling increment is obtained by expanding m channels; The additional downsampling module, the additional upsampling module and the super-resolution module are further used to perform additional downsampling based on an on-demand mechanism to obtain the next information increment and enter the reconstruction of the next incremental reconstruction round until the final reconstructed image meets the requirements, at which time the reconstruction is stopped.

6. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the processor executes the computer program, the steps of the image super-resolution method based on low-resolution increments according to any one of claims 1 to 4 are implemented.

Images