Image Rescaling

Abstract

According to an implementation of the subject matter described herein, a solution for image rescaling is proposed. According to this solution, an input image at a first resolution is obtained. Using a trained reversible neural network, an output image at a second resolution and high-frequency information according to a predetermined distribution are generated based on the input image, where the first resolution exceeds the second resolution. Furthermore, a further input image at the second resolution is obtained. Using an inverse network of the reversible neural network, a further output image at the first resolution is generated based on the further input image and the high-frequency information according to the predetermined distribution. This solution can downscale the original image to a visually pleasing low-resolution image with the same semantics, and can also reconstruct a high-quality high-resolution image from the low-resolution image.

Description

[Background technology]

[0001] background

[0001] Image rescaling has become one of the most common operations for digital image processing. On the one hand, the rapid increase in the amount of high-resolution (HR) images / videos on the Internet makes image downscaling extremely essential for storing, transferring and sharing such large data, because downscaling the counterparts can greatly save storage and utilize bandwidth efficiently while maintaining the same semantic information. On the other hand, many of these downscaling scenarios inevitably raise a large demand for the inverse task (i.e., upscaling the downscaled image to its original size).

[0002]

[0002] Conventional image downscaling (i.e., downscaling a high-resolution image to a low-resolution image) schemes usually lead to a loss of high-frequency information in the high-resolution image. Due to the loss of high-frequency information, conventional image upscaling (i.e., upscaling a low-resolution image to a high-resolution image) schemes often fail to reconstruct a high-quality high-resolution image from the low-resolution image. Summary of the Invention

[0003] overview

[0003] According to an implementation of the subject matter described herein, a solution for image rescaling is proposed. According to this solution, an input image of a first resolution is obtained. By using a trained reversible neural network, an output image of a second resolution and high-frequency information according to a predefined distribution are generated based on the input image, the first resolution exceeds the second resolution, and the input image and the output image have the same semantics. Moreover, a further input image of a second resolution is obtained. By using an inverse network of the reversible neural network, a further output image of the first resolution is generated based on the further input image and the high-frequency information according to a predefined distribution, and the further input image and the further output image have the same semantics. This solution can downscale the original image to a visually pleasant low-resolution image with the same semantics, and can also reconstruct a high-resolution image of high quality from the low-resolution image.

[0004]

[0004] This Summary is provided to introduce selected concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. [Brief description of the drawings]

[0005] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1A]

[0005] FIG. 1 illustrates a schematic block diagram of a computing device capable of implementing several implementations of the subject matter described herein. [Figure 1B]

[0006] 1 shows a schematic diagram of the working principle of an image rescaling module according to an implementation of the subject matter described herein. [Figure 2A]

[0007] FIG. 1 shows a schematic block diagram of a reversible neural network according to an implementation of the subject matter described herein. [Figure 2B]

[0008] 1 shows a schematic diagram of an example of a reversible neural network unit according to an implementation of the subject matter described herein. [Figure 3A]

[0009] 1 shows a schematic block diagram of an inverse network of a reversible neural network according to an implementation of the subject matter described herein. [Figure 3B]

[0010] 1 shows a schematic diagram of an example of a reversible neural network unit according to an implementation of the subject matter described herein. [Figure 4]

[0011] 1 shows a flowchart of an example method for image rescaling according to an implementation of the subject matter described herein. [Diagram 5]

[0012] 1 shows a flowchart of an example method for image rescaling according to an implementation of the subject matter described herein. [Figure 6]

[0013] 1 shows a block diagram of an example system in which implementations of the subject matter described herein can be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006]

[0014] Throughout the drawings, the same or similar reference symbols refer to the same or similar elements.

[0007] Detailed Description

[0015] The subject matter described herein will now be discussed with reference to several example implementations. It should be understood that these implementations are not intended to imply any limitation on the scope of the subject matter, but are discussed solely to enable those skilled in the art to better understand and implement the subject matter described herein.

[0008]

[0016] As used herein, the term "comprises" and variations thereof should be read as open-ended terms meaning "including, but not limited to." The term "based on" should be read as "based at least in part on." The terms "one implementation" and "an implementation" should be read as "at least one implementation." The term "another implementation" should be read as "at least one other implementation." The terms "first," "second," and similar terms may refer to different or the same subject matter. Other definitions may be included below, both explicitly and implicitly.

[0009]

[0017] As used herein, a "neural network" can handle an input and provide a corresponding output, and typically includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. Each layer of a neural network is connected in sequence such that the output of the previous layer is provided as the input of the next layer, and the input layer receives the input of the neural network model, and the output of the output layer is the final output of the neural network model. Each layer of a neural network model includes one or more nodes (also known as processing nodes or neurons), and each node processes the input from the previous layer. In this document, the terms "neural network", "model", "network" and "neural network model" can be used interchangeably.

[0010]

[0018] As described above, image rescaling is one of the most common operations for digital image processing. However, conventional image downscaling (i.e., downscaling a high-resolution image to a low-resolution image) schemes usually lead to the loss of high-frequency information in the high-resolution image. The loss of high-frequency information also makes image upscaling procedures (i.e., upscaling a low-resolution image to a high-resolution image) very challenging, because it means that the same low-resolution (LR) image may correspond to multiple high-resolution (HR) images (also known as ill-posedness in image upscaling procedures). Accordingly, conventional schemes usually fail to reconstruct high-quality HR images from LR images.

[0011]

[0019] Traditional schemes usually upscale LR images with super-resolution (SR) methods. Existing SR methods mainly focus on learning prior information through example-based strategies or deep learning models. Obviously, when a target LR image is obtained by downscaling the corresponding HR image, taking the image downscaling method into account during the image upscaling procedure is helpful in improving the quality of HR image reconstruction.

[0012]

[0020] Traditional image downscaling methods employ frequency-based kernels (such as bilinear and bicubic interpolation) as low-pass filters to subsample input HR images to a target resolution. However, the above methods often result in over-smoothed images because high-frequency information is suppressed. Recently, several detail-preserving or structurally similar downscaling methods have been proposed. However, these perceptually oriented downscaling methods never consider the potential mutual reinforcement between image downscaling and its inverse task (i.e., image upscaling).

[0013]

[0021] Inspired by the potential mutual reinforcement between image downscaling and its inverse task (i.e., image upscaling), some conventional schemes attempt to model image downscaling and image upscaling into a unified task. For example, some schemes provide an image downscaling model based on an autoencoder framework, in which the encoder and decoder act as image downscaling and SR models, respectively, so that the image downscaling and upscaling procedures are trained together as a unified task. Some schemes use convolutional neural networks to estimate the downscaled low-resolution image and utilize a learned or specified SR model for HR image reconstruction. Some schemes further propose content-adaptive sampler-based image downscaling methods, which can be trained together with any existing SR model. Although the above schemes can improve the quality of the HR image restored from the downscaled LR image to some extent, they cannot fundamentally solve the ill-posed problem of the image upscaling procedure, and thus fail to reconstruct a high-quality HR image from the LR image.

[0014]

[0022] According to an implementation of the subject matter described herein, a solution for image rescaling is provided. In this solution, an input image of a first resolution is rescaled to an output image of a second resolution by using a reversible neural network. Moreover, an inverse network of the neural network can rescale the input image of the second resolution to an output image of the first resolution. Specifically, during image downscaling, the reversible neural network can convert an HR image into an LR image and high-frequency noise following a predefined distribution. During image upscaling, the inverse network of the reversible neural network can convert an LR image and random noise following a predefined distribution into an HR image. Since the reversible neural network is used to model the image downscaling and upscaling procedures, this solution can significantly alleviate the ill-posed problem of the image upscaling procedure, so that the original image can be downscaled to a visually pleasant low-resolution image and a high-quality high-resolution image can be reconstructed from the low-resolution image.

[0015]

[0023] In the following, examples of different implementations of the solution are explained in more detail with reference to the drawings.

[0016]

[0024] FIG. 1A illustrates a block diagram of a computing device 100 capable of implementing multiple implementations of the subject matter described herein. It should be understood that the computing device 100 illustrated in FIG. 1 is merely exemplary and does not pose a limitation in the functionality and scope of the implementations described by the subject matter described herein. According to FIG. 1, the computing device 100 includes a computing device 100 in the form of a general-purpose computing device. Components of the computing device 100 may include, but are not limited to, one or more processors or processing units 110, memory 120, storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160.

[0017]

[0025] In some implementations, the computing device 100 can be implemented as various user terminals or service terminals having computing capabilities. The service terminals can be servers, large-scale computing devices, and the like provided by various service providers. The user terminals can be, for example, any type of mobile, fixed, or portable terminal, including mobile phones, mobile sites, mobile units, mobile devices, multimedia computers, multimedia tablets, Internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / videos, positioning devices, television sets, radio broadcast receivers, electronic book devices, game devices, or any other combinations thereof (consisting of accessories and peripherals of these devices, or any other combinations thereof). It can also be envisioned that the computing device 100 can support any type of user-specific interface (such as "wearable" circuits and the like).

[0018]

[0026] The processing unit 110 may be a physical or virtual processor and may perform various operations based on programs stored in the memory 120. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of the computing device 100. The processing unit 110 may also be known as a central processing unit (CPU), a microprocessor, a controller, and a microcontroller.

[0019]

[0027] Computing device 100 typically includes a number of computer storage media. Such media may be any attainable media accessible by computing device 100, including but not limited to, volatile and nonvolatile media, removable and non-removable media. Memory 120 may be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash, etc.), or any combination thereof.

[0020]

[0028] Storage device 130 may be removable or non-removable media and may include a machine-readable medium (e.g., memory, flash drive, magnetic disk) or any other medium that can be used to store information and / or data and that can be accessed within computing device 100. Computing device 100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 1, a disk drive for reading from or writing to a removable non-volatile disk and an optical disk drive for reading from or writing to a removable non-volatile optical disk may be provided. In such cases, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0021]

[0029] The communication unit 140 facilitates communication with another computing device through a communication medium. In addition, the functionality of the components of the computing device 100 may be realized by a single computing cluster or multiple computing machines, which may communicate through a communication connection. Thus, the computing device 100 may operate in a networked environment using logical connections with one or more other servers, personal computers (PCs), or further general-purpose network nodes.

[0022]

[0030] The input device 150 may be one or more of various input devices, such as a mouse, a keyboard, a trackball, a voice input device, and the like. The output device 160 may be one or more output devices, such as, for example, a display, a loudspeaker, and a printer. The computing device 100 may also communicate with one or more external devices (not shown) through the communication unit 140, as needed, where the external devices (e.g., storage devices, display devices, etc.) communicate with one or more devices for enabling a user to interact with the computing device 100, or with any device (such as a network card, modem, and the like) for enabling the computing device 100 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0023]

[0031] In some implementations, apart from being integrated on individual devices, some or all of the respective components of the computing device 100 may also be configured in the form of a cloud computing architecture. In a cloud computing architecture, these components may be located remotely and may cooperate to implement the functionality described by the subject matter described herein. In some implementations, cloud computing provides computing, software, data access, and storage services without the physical location or configuration of the systems or hardware providing such services being made known to the terminal user. In various implementations, cloud computing provides services over a wide area network (such as the Internet) using an appropriate protocol. For example, a cloud computing provider provides applications that can be accessed through a web browser or any other computing component over the wide area network. The software or components of the cloud computing architecture and corresponding data may be stored on a server at a remote location. The computing resources of the cloud computing environment may be merged or distributed at a remote data center. The cloud computing infrastructure may provide services through a shared data center even when presented as a single access point to the user. Thus, the components and functions described herein may be provided using a cloud computing architecture from a service provider at a remote location. Alternatively, the components and functionality may be provided from a traditional server, or may reside directly or otherwise on the client device.

[0024]

[0032] The computing device 100 can be used to perform image rescaling in accordance with various implementations of the subject matter described herein. The memory 120 can include an image rescaling module 122 having one or more program instructions. The module can be accessed and operated by the processing unit 110 to implement the functions of the various implementations described herein.

[0025]

[0033] During image rescaling, the computing device 100 can receive an input image 170 via the input device 150. In some implementations, the input image 170 can be, for example, an image of a first resolution. The input image 170 can be input to the image rescaling module 122 in the memory 120. The image rescaling module 122 can use a trained reversible neural network to generate an output image 180 of a second resolution and high-frequency information following a predefined distribution based on the input image 170, where the first resolution exceeds the second resolution, and the input image 170 and the output image 180 have the same semantics. In other implementations, the input image 170 can be, for example, an image of a second resolution. The input image 170 can be input to the image rescaling module 122 in the memory 120. The image rescaling module 122 can use an inverse network of a reversible neural network to generate an output image 180 of a first resolution based on the input image 170 and high-frequency information following a predefined distribution, where the first resolution exceeds the second resolution, and the input image 170 and the output image 180 have the same semantics. The output image 180 can be output via the output device 160.

[0026]

[0034] In some implementations, the image rescaling module 122 can perform image downscaling (i.e., converting an HR image to an LR image) using a trained reversible neural network, and the image rescaling module 122 can perform inverse image upscaling (i.e., reconstructing an LR image to an HR image) using an inverse network of the reversible neural network. FIG. 1B shows a schematic diagram of the working principle of the image rescaling module 122 according to an implementation of the subject matter described herein. As shown, the image rescaling module 122 uses a reversible neural network 191 ("f θ ") can be used to generate a low-resolution output image 180 and high-frequency information 185 that follows a predefined distribution based on a high-resolution input image 170. For example, the high-frequency information 185 can be embodied as high-frequency noise that is unrelated to the semantics of the input image 170. The image rescaling module 122 uses an inverse network 192 (

number

[0027]

[0035] Reversible Neural Networks (INNs) are a common network structure in generative models, with a mapping relationship m=f θ (n) and its inverse mapping relationship

number

number

number

number

number

number

number

[0028]

[0036] When the INN is applied to the image rescaling task, based on a high-resolution input image x, the INN can output a downscaled low-resolution image y and high-frequency information z that follows a predefined distribution, where the high-frequency information z can be embodied, for example, as high-frequency noise that is unrelated to the semantics of the image. In this way, the inverse network of the INN can reconstruct a high-quality high-resolution image x based on the low-resolution image y and the noise z. In other words, to make the image rescaling procedure reversible, it is necessary to maintain the high-frequency information z that is usually lost in the image downscaling procedure. Moreover, the whole image rescaling procedure is based on the mapping relationship (y,z)=f θ (x), and

number

[0029]

[0037] However, during image upscaling, usually any LR image needs to be upscaled. Therefore, the high frequency information z corresponding to the input LR image is often not present. The inventors have realized that the information lost in the image downscaling procedure is equivalent to high frequency details according to the Nyquist-Shannon sampling theory. Assuming that a group of HR images corresponding to the same LR image contain different high frequency details, these details may usually exhibit some degree of variability and randomness. Therefore, z can be represented as a random variable, whose distribution depends on how the INN represents z (i.e., f θ output z). Specifically, the INN can be trained to satisfy a predefined distribution p(z). In this way, there is no need to preserve the high-frequency noise z output by the reversible neural network during the image downscaling procedure. In addition, during the image upscaling procedure, the high-resolution image can be reconstructed based on the low-resolution image and any one sample under the predefined distribution.

[0030]

[0038] 2A shows a schematic block diagram of a reversible neural network 191 according to an implementation of the subject matter described herein. It should be understood that the structure of the reversible neural network 191 shown in FIG. 2A is merely exemplary and does not imply any limitation on the scope of the subject matter described herein. Implementations of the subject matter described herein are also suitable for reversible neural networks having different structures.

[0031]

[0039] As shown in Fig. 2A, the reversible neural network 19 can be formed by connecting one or more downsampling modules 210 in series. For simplicity, only one downsampling module 210 is shown in Fig. 2A. The image downscaling ratio supported by the reversible neural network 191 can be determined by the image downscaling ratio supported by each downsampling module 210 and the number of downsampling modules 210 included in the reversible neural network 191. For example, assuming that each downsampling module 210 supports 1 / 2-fold image reduction and the reversible neural network 191 includes two downsampling modules 210, the reversible neural network 191 supports 1 / 4-fold image reduction.

[0032]

[0040] As shown in FIG. 2A, for example, the downsampling module 210 may include a conversion module 230 and one or more INN units 220-1, 220-2...220-M (collectively known as "INN units 220" or individually as "INN units 220", where M≧1).

[0033]

[0041] The transform module 230 can decompose the high-resolution input image 170 into low-frequency components 242 that represent the semantics of the input image 170, and high-frequency components 241 that are related to the semantics. In some implementations, the transform module 230 can be implemented as a wavelet transform module (e.g., a Haar transform module). For example, when the transform module 230 is implemented as a Haar transform module, the downsampling module 210 can support ½-fold reduction of the image. Specifically, the Haar transform module converts a collection of input images or feature maps with length H, width W, and number of channels C into an output tensor

number

[0034]

[0042] As explained above, to ensure that the network structure of the neural network 191 is reversible, the structure of each INN unit 220 should be reversible. The INN unit 220 is used to extract corresponding features from the input low-frequency and high-frequency components, and convert the high-frequency components related to image semantics into high-frequency information, which follows a predetermined distribution and is independent of image semantics.

[0035]

[0043] 2B illustrates a schematic diagram of an example of an INN unit 220 according to an implementation of the subject matter described herein. Here, the low and high frequency components input to the INN unit 220 are

number

number

number

number

[0036]

[0044] 3A shows a schematic block diagram of the inverse network 192 of the reversible neural network 191 of FIG. 2A. As shown in FIG. 3A, the network 192 can be formed by connecting one or more upsampling modules 310 in series. For simplicity, only one upsampling module 310 is shown in FIG. 3A. The image upscaling ratio supported by the inverse network 192 can be determined by the image upscaling ratio supported by each upsampling module 310 and the number of upsampling modules 310 included in the inverse network 192. For example, assuming that each upsampling module 310 supports 2x image enlargement and the inverse network 192 includes two upsampling modules 310, the inverse network 192 supports 4x image enlargement.

[0037]

[0045] As shown in FIG. 3A, for example, the upsampling module 310 may include a transform module 330 and one or more INN units 320-1, 320-2...320-M (collectively known as "INN units 320" or individually as "INN units 320", where M > 1). For example, the structure of the INN unit 320 and the structure of the INN unit 220 of FIG. 2B are invertible to each other, as shown in FIG. 3B. For the INN unit 320-M, the low-resolution input image 170 input to the INN unit 320-M is

number

number

number

number

[0038]

[0046] As shown in FIG. 3A, one or more INN units 320 can transform the low-resolution input image 170 and the high-frequency information 175 according to a predetermined distribution into a combined high-frequency component 341 and a low-frequency component 342. Contrary to the transform module 230 shown in FIG. 2A, the transform module 330 can combine the high-frequency component 341 and the low-frequency component 342 into a high-resolution output image 180. In some implementations, when the transform module 230 is implemented as a wavelet transform module, the transform module 330 can be implemented as an inverse wavelet transform module. For example, when the transform module 230 is implemented as a Haar transform module, the transform module 330 can be implemented as an inverse Haar transform module. Alternatively, the transform module 330 can be implemented by a 1×1 reversible convolution block or any currently known or future developed transform module that can combine the high-frequency component and the low-frequency component into an image.

[0039]

[0047] The training procedure of the reversible neural network is described in more detail below. In this document, the neural network to be trained and its inverse network are collectively known as the "model" for simplicity. According to the above description, the purpose of training the model is to find a mapping relationship f between the high-resolution image x, the low-resolution image y, and a predefined distribution p(z). θ It can be seen that the purpose of the present invention is to determine the following:

[0040]

[0048] To achieve training purposes, in some implementations, a set of high-resolution images is

number

[0041]

[0049] In some implementations, the objective function for training the model can be determined based on the difference between the low-resolution training image and the low-resolution image generated by the model based on the high-resolution training image. For example, a high-resolution training image x of the first group of training images can be (n) For high-resolution training image x (n) The low-resolution images generated by the model based on

number

number

number

number

number

number

[0042]

[0050] Additionally or alternatively, in some implementations, the objective function for training the model can be determined based on the difference between the high-resolution training image and the high-resolution image reconstructed by the model based on the low-resolution image. For example, a high-resolution training image x (n) For high-resolution training image x (n) The low-resolution images generated by the model based on

number

number

number

number

number

number

number

number

[0043]

[0051] Additionally or alternatively, another objective of model training is to encourage the model to capture the data distribution of the high-resolution training images. Here, the first set of training data

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0044]

[0052] In some cases, it may be difficult to directly minimize the third objective function shown in equation (6) because the two distributions are both high-dimensional and have unknown density functions. In some implementations, the JS divergence can be used to measure the difference between the two data distributions. That is, the third objective function can also be expressed as follows:

number

[0045]

[0053] In some implementations, a total objective function for training a model can be generated by combining the first objective function, the second objective function, and the third objective function. For example, the total objective function can be expressed as follows: L total : λ1L recon +λ2L guide +λ3L distr (8) where λ1, λ2 and λ3 are coefficients to balance the different loss terms.

[0046]

[0054] In some implementations, to improve the stability of model training, a pre-training stage can be performed before training the model with the full objective function shown in equation (8). In the pre-training stage, a weaker but more stable distribution-matched loss function can be adopted. For example, the distribution-matched loss function can be constructed based on the cross-entropy loss function to enhance the stability of model training. For example, a distribution-matched loss function (also known as the "fourth objective function") constructed based on the cross-entropy (CE) loss function can be expressed as follows:

number

number

[0047]

[0055] In some implementations, after the pre-training phase, a second training may be performed on the model, again based on the overall objective function shown in equation (8). Alternatively, in some implementations, after the pre-training phase, a second training may be performed on the model, again based on the overall objective function shown in equation (11), as follows: L IRN+ :=λ1L recon +λ2L guide +λ3L distr +λ4L percp (11) where L is a perceptual loss function to measure the difference between the original high-resolution image and the reconstructed high-resolution image in semantic features. percpFor example, the semantic features of the original high-resolution image and the reconstructed high-resolution image can be extracted by benchmark models known in the art, which will not be detailed here. λ1, λ2, λ3 and λ4 are coefficients for balancing different loss terms.

[0048]

[0056] 4 shows a flowchart of a method 400 for image rescaling according to some implementations of the subject matter described herein. The method 400 can be implemented by the computing device 100, for example in the image rescaling module 122 in the memory 120 of the computing device 100. In block 410, the computing device 100 obtains an input image of a first resolution. In block 420, the computing device 100 uses a trained reversible neural network to generate an output image of a second resolution and high-frequency information according to a predefined distribution based on the input image, where the first resolution exceeds the second resolution, and the input image and the output image have the same semantics.

[0049]

[0057] In some implementations, the reversible neural network includes a transformation module and at least one reversible network unit, and generating the output image and the high-frequency information includes using the transformation module to decompose the input image into low-frequency components representing the semantics of the input image and high-frequency components related to the semantics, and using the at least one reversible network unit to generate the output image and the high-frequency information unrelated to the semantics based on the low-frequency components and the high-frequency components.

[0050]

[0058] In some implementations, the transform module includes one of a wavelet transform module and a reversible convolution block.

[0051]

[0059] In some implementations, the method 400 further includes training a reversible neural network, where the reversible neural network is trained to generate a second image at a second resolution and the first high-frequency information according to a predetermined distribution based on the first image at the first resolution, and an inverse network of the reversible neural network is trained to generate a fourth image at the first resolution based on the third image at the second resolution and the second high-frequency information according to the predetermined distribution.

[0052]

[0060] In some implementations, training the reversible neural network includes obtaining a first group of training images at a first resolution, obtaining a second group of training images at a second resolution that respectively correspond to semantics of the first group of training images, and training the reversible neural network based on the first group of training images and the second group of training images.

[0053]

[0061] In some implementations, obtaining the second group of training images includes generating the second group of training images based on the first group of training images and using an interpolation method.

[0054]

[0062] In some implementations, training the reversible neural network includes determining a plurality of objective functions based on a first group of training images and a second group of training images, determining an overall objective function for training the reversible neural network by combining at least a portion of the plurality of objective functions, and determining network parameters of the reversible neural network by minimizing the overall objective function.

[0055]

[0063] In some implementations, determining the multiple objective functions includes generating a third group of training images at a second resolution and a group of random variables based on the first group of training images and using a reversible neural network, and determining a first objective function based on a difference between the second group of training images and the third group of training images.

[0056]

[0064] In some implementations, determining the multiple objective functions includes generating a third group of training images at a second resolution and a group of random variables based on the first group of training images and using a reversible neural network, generating a fourth group of training images at a first resolution based on the third group of training images and high-frequency information following a predefined distribution using the inverse network, and determining a second objective function based on a difference between the first group of training images and the fourth group of training images.

[0057]

[0065] In some implementations, determining the multiple objective functions includes determining a first data distribution for the first group of training images, determining a second data distribution for the fourth group of training images, and determining a third objective function based on a difference between the first data distribution and the second data distribution.

[0058]

[0066] In some implementations, determining the multiple objective functions includes determining a third data distribution for the group of random variables and determining a fourth objective function based on a difference between the third data distribution and the predetermined distribution.

[0059]

[0067] 5 shows a flowchart of a method 500 for image rescaling according to some implementations of the subject matter described herein. The method 500 can be implemented by the computing device 100, for example in the image rescaling module 122 in the memory 120 of the computing device 100. In block 510, the computing device 100 obtains an input image of a second resolution. In block 520, the computing device 100 uses a trained reversible neural network to generate an output image of a first resolution based on the input image and high-frequency information following a predefined distribution, where the first resolution exceeds the second resolution, and the input image and the output image have the same semantics.

[0060]

[0068] In some implementations, the reversible neural network includes a transformation module and at least one reversible network unit, and generating the output image includes using the at least one reversible network unit to generate combined low-frequency and high-frequency components based on the input image and high-frequency information, where the low-frequency components represent semantics of the input image and the high-frequency components are related to the semantics, and combining the low-frequency and high-frequency components into the output image using the transformation module.

[0061]

[0069] In some implementations, the transform module includes one of a wavelet transform module and a reversible convolution block.

[0062]

[0070] In light of the above, the implementation of the subject matter described herein proposes a solution for image rescaling. During image downscaling, a reversible neural network can convert an HR image into an LR image and high-frequency noise following a predefined distribution. During image upscaling, an inverse network of the reversible neural network can convert an LR image and random noise following a predefined distribution into an HR image. Since a reversible neural network is used to model image downscaling and upscaling procedures, the solution can downscale an original image into a visually pleasant low-resolution image and reconstruct a high-quality high-resolution image from the low-resolution image, thereby greatly alleviating the ill-posed problem of the image upscaling procedure. Moreover, various experimental data have also shown that compared with conventional image rescaling schemes, the implementation of the subject matter described herein can achieve better image reconstruction performance indicators, such as higher peak signal-to-noise ratio (PSNR) and / or structural similarity (SSIM).

[0063]

[0071] Implementations of the subject matter described herein can be widely applied to image and / or video processing fields. For example, online video streaming plays an important role in people's lives, such as video sites, live streaming sites, video streaming mobile applications, etc. High-quality online video streaming, such as high-resolution video with rich perceptual details, is desirable. However, high-resolution video usually requires a lot of network bandwidth for transmission. Therefore, to save network bandwidth, high-resolution video is usually processed and compressed before being sent to a user client. This results in poor quality low-resolution video being presented at the user client side. The above problems can be solved by applying an image rescaling solution according to implementations of the subject matter described herein.

[0064]

[0072] 6 shows a block diagram of an example system 600 in which implementations of the subject matter described herein can be implemented. As shown, the system 600 can include a video stream service provider 610, a server 620, and a client device 630. For example, the video stream service provider 610 can provide video data requested by the client device 630 to the server 620, and the server 620 can transmit the video data from the video stream service provider 610 to the client device 630 over a network.

[0065]

[0073] As shown in FIG. 6, in some implementations, a video stream service provider 610 can provide a high-resolution video stream 601 to a server 620, where the high-resolution video stream 601 is also known as a “high-resolution image sequence 601.” The server 620 can convert the high-resolution image sequence 601 into a low-resolution image sequence using a reversible neural network 191 as described above. In some implementations, the server 620 can directly transmit the low-resolution image sequence to a client device 630 as a low-resolution video stream 602. In this case, the client device 630 can receive the low-resolution image sequence. Additionally or alternatively, in some implementations, the server 620 can perform video encoding on the low-resolution image sequence to generate an encoded low-resolution video stream 602 and transmit the encoded low-resolution video stream 602 to the client device 630 over a network. In this case, the client device 630 can decode the received encoded low-resolution video stream 602 to derive the decoded low-resolution image sequence. The client device 630 can then use the inverse network 192 of the reversible neural network 191 to reconstruct the derived low-resolution image sequence into a high-resolution video stream 603. In this way, the client can obtain a high-quality video stream while saving network bandwidth.

[0066]

[0074] In addition to the image and / or video processing field, implementations of the subject matter described herein can also be applied to the image and / or video storage field. For example, before storing the high-resolution image and / or video in a storage device, the reversible neural network 191 as described above can be used to convert the high-resolution image and / or video into a low-resolution image and / or video and corresponding high-frequency information according to a predetermined distribution. The derived low-resolution image and / or video can then be stored in the storage device, while the corresponding high-frequency information can be discarded. To access the image and / or video stored in the storage device, the low-resolution image and / or video can first be obtained from the storage device. Then, the inverse network 192 of the reversible neural network 191 as described above can be used to reconstruct the high-resolution image and / or video based on the obtained low-resolution image and / or video and random noise according to a predetermined distribution. In this manner, storage space for storing the image and / or video can be saved without compromising the quality of the image and / or video.

[0067]

[0075] Below are listed some examples of implementations of the subject matter described in this specification.

[0068]

[0076] In a first aspect, the subject matter described herein provides a computer-implemented method, the method including obtaining an input image of a first resolution, and generating an output image of a second resolution and high-frequency information following a predefined distribution based on the input image and using a trained reversible neural network, where the first resolution exceeds the second resolution, and the input image and the output image have the same semantics.

[0069]

[0077] In some implementations, the method further includes storing the output image without storing high frequency information.

[0070]

[0078] In some implementations, the method further includes encoding the output image and providing an encoded output image.

[0071]

[0079] In some implementations, the reversible neural network includes a transformation module and at least one reversible network unit, and generating the output image and the high-frequency information includes using the transformation module to decompose the input image into low-frequency components representing the semantics of the input image and high-frequency components related to the semantics, and using the at least one reversible network unit to generate the output image and the high-frequency information unrelated to the semantics based on the low-frequency components and the high-frequency components.

[0072]

[0080] In some implementations, the transform module includes one of a wavelet transform module and a reversible convolution block.

[0073]

[0081] In some implementations, the method further includes training a reversible neural network, where the reversible neural network is trained to generate a second image at a second resolution and the first high-frequency information according to a predetermined distribution based on the first image at the first resolution, and an inverse network of the reversible neural network is trained to generate a fourth image at the first resolution based on the third image at the second resolution and the second high-frequency information according to the predetermined distribution.

[0074]

[0082] In some implementations, training the reversible neural network includes obtaining a first group of training images at a first resolution, obtaining a second group of training images at a second resolution that respectively correspond to semantics of the first group of training images, and training the reversible neural network based on the first group of training images and the second group of training images.

[0075]

[0083] In some implementations, obtaining the second group of training images includes generating the second group of training images based on the first group of training images and using an interpolation method.

[0076]

[0084] In some implementations, training the reversible neural network includes determining a plurality of objective functions based on a first group of training images and a second group of training images, determining an overall objective function for training the reversible neural network by combining at least a portion of the plurality of objective functions, and determining network parameters of the reversible neural network by minimizing the overall objective function.

[0077]

[0085] In some implementations, determining the multiple objective functions includes generating a third group of training images at a second resolution and a group of random variables based on the first group of training images and using a reversible neural network, and determining a first objective function based on a difference between the second group of training images and the third group of training images.

[0078]

[0086] In some implementations, determining the multiple objective functions includes generating a third group of training images at a second resolution and a group of random variables based on the first group of training images and using a reversible neural network, generating a fourth group of training images at a first resolution based on the third group of training images and high-frequency information following a predefined distribution using the inverse network, and determining a second objective function based on a difference between the first group of training images and the fourth group of training images.

[0079]

[0087] In some implementations, determining the multiple objective functions includes determining a first data distribution for the first group of training images, determining a second data distribution for the fourth group of training images, and determining a third objective function based on a difference between the first data distribution and the second data distribution.

[0080]

[0088] In some implementations, determining the multiple objective functions includes determining a third data distribution for the group of random variables and determining a fourth objective function based on a difference between the third data distribution and the predetermined distribution.

[0081]

[0089] In a second aspect, the subject matter described herein provides a computer-implemented method, the method including obtaining an input image at a second resolution, and using a trained reversible neural network to generate an output image at a first resolution based on the input image and high-frequency information following a predefined distribution, the first resolution being greater than the second resolution, and the input image and the output image having the same semantics.

[0082]

[0090] In some implementations, obtaining the input image includes obtaining an encoded input image and decoding the encoded input image.

[0083]

[0091] In some implementations, the reversible neural network includes a transformation module and at least one reversible network unit, and generating the output image includes using the at least one reversible network unit to generate combined low-frequency and high-frequency components based on the input image and high-frequency information, where the low-frequency components represent semantics of the input image and the high-frequency components are related to the semantics, and combining the low-frequency and high-frequency components into the output image using the transformation module.

[0084]

[0092] In some implementations, the transform module includes one of a wavelet transform module and a reversible convolution block.

[0085]

[0093] In a third aspect, the subject matter described herein provides an electronic device, the electronic device including a processing unit and a memory coupled to the processing unit, the memory including instructions stored thereon, which when executed by the processing unit cause the device to perform acts including obtaining an input image at a first resolution, and generating, based on the input image and using a trained reversible neural network, an output image at a second resolution and high frequency information following a predefined distribution, where the first resolution exceeds the second resolution, and the input image and the output image have the same semantics.

[0086]

[0094] In some implementations, the acts further include storing the output image without storing the high frequency information.

[0087]

[0095] In some implementations, the acts further include encoding the output image and providing an encoded output image.

[0088]

[0096] In some implementations, the reversible neural network includes a transformation module and at least one reversible network unit, and generating the output image and the high-frequency information includes using the transformation module to decompose the input image into low-frequency components representing the semantics of the input image and high-frequency components related to the semantics, and using the at least one reversible network unit to generate the output image and the high-frequency information unrelated to the semantics based on the low-frequency components and the high-frequency components.

[0089]

[0097] In some implementations, the transform module includes one of a wavelet transform module and a reversible convolution block.

[0090]

[0098] In some implementations, the acts further include training a reversible neural network, where the reversible neural network is trained to generate a second image at a second resolution and the first high-frequency information according to a predetermined distribution based on the first image at the first resolution, and an inverse network of the reversible neural network is trained to generate a fourth image at the first resolution based on the third image at the second resolution and the second high-frequency information according to the predetermined distribution.

[0091]

[0099] In some implementations, training the reversible neural network includes obtaining a first group of training images at a first resolution, obtaining a second group of training images at a second resolution that respectively correspond to semantics of the first group of training images, and training the reversible neural network based on the first group of training images and the second group of training images.

[0092]

[0100] In some implementations, obtaining the second group of training images includes generating the second group of training images based on the first group of training images and using an interpolation method.

[0093]

[0101] In some implementations, training the reversible neural network includes determining a plurality of objective functions based on a first group of training images and a second group of training images, determining an overall objective function for training the reversible neural network by combining at least a portion of the plurality of objective functions, and determining network parameters of the reversible neural network by minimizing the overall objective function.

[0094]

[0102] In some implementations, determining the multiple objective functions includes generating a third group of training images at a second resolution and a group of random variables based on the first group of training images and using a reversible neural network, and determining a first objective function based on a difference between the second group of training images and the third group of training images.

[0095]

[0103] In some implementations, determining the multiple objective functions includes generating a third group of training images at a second resolution and a group of random variables based on the first group of training images and using a reversible neural network, generating a fourth group of training images at a first resolution based on the third group of training images and high-frequency information following a predefined distribution using the inverse network, and determining a second objective function based on a difference between the first group of training images and the fourth group of training images.

[0096]

[0104] In some implementations, determining the multiple objective functions includes determining a first data distribution for the first group of training images, determining a second data distribution for the fourth group of training images, and determining a third objective function based on a difference between the first data distribution and the second data distribution.

[0097]

[0105] In some implementations, determining the multiple objective functions includes determining a third data distribution for the group of random variables and determining a fourth objective function based on a difference between the third data distribution and the predetermined distribution.

[0098]

[0106] In a fourth aspect, the subject matter described herein provides an electronic device, the electronic device including a processing unit and a memory coupled to the processing unit, the memory including instructions stored thereon, which when executed by the processing unit cause the device to perform acts including obtaining an input image at a second resolution and using a trained reversible neural network to generate an output image at a first resolution based on the input image and high frequency information according to a predefined distribution, the first resolution being greater than the second resolution, and the input image and the output image having the same semantics.

[0099]

[0107] In some implementations, obtaining the input image includes obtaining an encoded input image and decoding the encoded input image.

[0100]

[0108] In some implementations, the reversible neural network includes a transformation module and at least one reversible network unit, and generating the output image includes using the at least one reversible network unit to generate combined low-frequency and high-frequency components based on the input image and high-frequency information, where the low-frequency components represent semantics of the input image and the high-frequency components are related to the semantics, and combining the low-frequency and high-frequency components into the output image using the transformation module.

[0101]

[0109] In some implementations, the transform module includes one of a wavelet transform module and a reversible convolution block.

[0102]

[0110] In a fifth aspect, the subject matter described herein provides a computer program product tangibly stored on a non-transitory computer storage medium and including machine-executable instructions that, when executed by a device, cause the device to perform a method according to the first or second aspect.

[0103]

[0111] In a further aspect, the subject matter described herein provides a computer-readable medium having machine-executable instructions stored thereon, the machine-executable instructions, when executed by a device, causing the device to perform a method according to the first or second aspect.

[0104]

[0112] The functionality described herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0105]

[0113] Program codes for implementing the subject methods described herein can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program codes cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The program codes can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0106]

[0114] In the context of the subject matter described herein, a machine-readable medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0107]

[0115] Furthermore, although operations are depicted in a particular order, it should be understood that the operations need not be performed in the particular order or sequence shown, or that all of the operations shown need to be performed to achieve the expected results. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although details of some specific implementations are included in the above discussion, these should not be construed as limiting the scope of the subject matter described herein. Certain features that are described in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable subcombination.

[0108]

[0116] Although the subject matter described herein has been described in language specific to structural features and / or method acts, it is to be understood that the subject matter described herein as specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. 1. A computer-implemented method, comprising: Obtaining an input image at a first resolution; generating an output image of a second resolution based on the input image and using a trained reversible neural network, the high frequency information being in accordance with a predetermined distribution; wherein the first resolution exceeds the second resolution, and the input image and the output image have the same semantics.

2. The reversible neural network includes a transformation module and at least one reversible network unit, and generating the output image and the high-frequency information includes: using the transformation module to decompose the input image into low frequency components representative of the semantics of the input image and high frequency components related to the semantics; generating the output image and the semantics-independent high-frequency information based on the low-frequency components and the high-frequency components using the at least one reversible network unit; The method of claim 1 , comprising:

3. The conversion module includes: Wavelet transform module, and Reversible Convolution Block The method of claim 2, comprising any one of the following:

4. training the reversible neural network; The reversible neural network is trained to generate, based on a first image at the first resolution, a second image at the second resolution and first high-frequency information according to the predetermined distribution; and 2. The method of claim 1, wherein the inverse network of the reversible neural network is trained to generate a fourth image in the first resolution based on a third image in the second resolution and second high-frequency information according to the predetermined distribution.

5. Training the reversible neural network comprises: obtaining a first set of training images at the first resolution; obtaining a second set of training images at the second resolution, the second set corresponding respectively to semantics of the first set of training images; training the reversible neural network based on the first group of training images and the second group of training images; The method of claim 4 , comprising:

6. Obtaining the second group of training images comprises: generating a second set of training images based on the first set of training images and using an interpolation method; The method of claim 5 , comprising:

7. Training the reversible neural network comprises: determining a plurality of objective functions based on the first group of training images and the second group of training images; determining an overall objective function for training the reversible neural network by combining at least a portion of the objective functions; determining network parameters of the reversible neural network by minimizing the overall objective function; The method of claim 5 , comprising:

8. Determining the plurality of objective functions comprises: generating a third set of training images at the second resolution and a set of random variables based on the first set of training images and using the reversible neural network; determining a first objective function based on a difference between the second group of training images and the third group of training images; The method of claim 7, comprising:

9. Determining the plurality of objective functions comprises: generating a third set of training images at the second resolution and a set of random variables based on the first set of training images and using the reversible neural network; generating a fourth set of training images at the first resolution based on the third set of training images and high frequency information according to the predetermined distribution using the inverse network; determining a second objective function based on differences between the first group of training images and the fourth group of training images; The method of claim 7, comprising:

10. Determining the plurality of objective functions comprises: determining a first data distribution for a first group of training images; determining a second data distribution for a fourth group of training images; and determining a third objective function based on a difference between the first data distribution and the second data distribution; and 10. The method of claim 9, comprising:

11. Determining the plurality of objective functions comprises: determining a third data distribution for the set of random variables; determining a fourth objective function based on a difference between the third data distribution and the predetermined distribution; and 10. The method of claim 9, comprising:

12. 1. A computer-implemented method, comprising: obtaining an input image at a second resolution; generating an output image at a first resolution based on the input image and high frequency information according to a predetermined distribution using a trained reversible neural network; wherein the first resolution exceeds the second resolution, and the input image and the output image have the same semantics.

13. The reversible neural network includes a transformation module and at least one reversible network unit, and generating the output image includes: generating, using the at least one reversible network unit, combined low-frequency and high-frequency components based on the input image and the high-frequency information, the low-frequency components representing semantics of the input image and the high-frequency components related to the semantics; using the transformation module to combine the low frequency components and the high frequency components into the output image; The method of claim 12, comprising:

14. The conversion module includes: Wavelet transform module, and Reversible Convolution Block 14. The method of claim 13, comprising any one of the following:

15. 1. An electronic device comprising: A processing unit; a memory coupled to the processing unit, the memory including instructions stored thereon; the instructions, when executed by the processing unit, Obtaining an input image at a first resolution; generating an output image of a second resolution based on the input image and using a trained reversible neural network, the high frequency information being in accordance with a predetermined distribution; causing the device to perform actions including The first resolution exceeds the second resolution, and the input image and the output image have the same semantics.

Images