Super-resolution image reconstruction

Abstract

According to implementations of the present disclosure, a scheme for super-resolution image reconstruction is proposed. According to the scheme, an input image having a first resolution is obtained. A reversible neural network is trained with the input image, wherein the reversible neural network is configured to generate an intermediate image having a second resolution and first high-frequency information based on the input image, and the second resolution is lower than the first resolution. Subsequently, an output image having a third resolution is generated based on the input image and second high-frequency information subject to a predetermined distribution by using an inverse network of the trained reversible neural network, wherein the third resolution is higher than the first resolution. The scheme can effectively process a low-resolution image obtained by an unknown downsampling method, thereby obtaining a high-quality high-resolution image.

Description

Background Technology

[0001] Super-resolution (SR) image reconstruction refers to the process of generating a high-resolution image from a low-resolution image. By increasing the resolution of an image, super-resolution technology can improve its quality, thus providing images with a clearer visual experience or enabling them to be better used for subsequent image processing tasks such as image analysis. Super-resolution technology has been applied to various aspects of people's lives.

[0002] With the development of computer technology, in recent years, in addition to traditional interpolation-based and reconstruction-based methods, machine learning-based methods have also been proposed. However, traditional machine learning methods rely on pairs of high-resolution and low-resolution images. Since such image pairs often do not exist in reality, some traditional schemes generate low-resolution images using known downsampling methods to create image pairs for training machine learning models. However, machine learning models trained with such image pairs typically only achieve good results when processing low-resolution images obtained using the same downsampling method, and struggle to effectively process low-resolution images obtained using other downsampling methods. Summary of the Invention

[0003] According to an implementation of this disclosure, a scheme for super-resolution image reconstruction is proposed. According to the scheme, an input image with a first resolution is acquired. An invertible neural network is trained using the input image, wherein the invertible neural network is configured to generate an intermediate image with a second resolution and first high-frequency information based on the input image, and the second resolution is lower than the first resolution. Subsequently, using the inverse network of the trained invertible neural network, an output image with a third resolution is generated based on the input image and the second high-frequency information following a predetermined distribution, wherein the third resolution is higher than the first resolution. This scheme can effectively process low-resolution images obtained by unknown downsampling methods, thereby obtaining high-quality high-resolution images.

[0004] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0005] Figure 1A A schematic block diagram of a computing device capable of implementing multiple implementations of the present disclosure is shown;

[0006] Figure 1B A schematic diagram illustrating the working principle of an image processing module implemented according to this disclosure is shown;

[0007] Figure 2A A schematic block diagram of a reversible neural network implemented according to this disclosure is shown;

[0008] Figure 2B A schematic diagram of an example reversible neural network unit according to an implementation of this disclosure is shown;

[0009] Figure 3 A schematic diagram of a training reversible neural network according to an implementation of the present disclosure is shown;

[0010] Figure 4A A schematic block diagram of the inverse network of a reversible neural network implemented according to the present disclosure is shown;

[0011] Figure 4B A schematic diagram of an example reversible neural network unit according to an implementation of this disclosure is shown; and

[0012] Figure 5 A flowchart of an example method for super-resolution image reconstruction according to an implementation of this disclosure is shown.

[0013] In these accompanying figures, the same or similar reference symbols are used to indicate the same or similar elements. Detailed Implementation

[0014] This disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement this disclosure, and not to imply any limitation on the scope of this disclosure.

[0015] As used herein, the term "comprising" and its variations are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "at least partially based on". The terms "an implementation" and "an implementation" are to be interpreted as "at least one implementation". The term "another implementation" is to be interpreted as "at least one other implementation". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0016] As used herein, a "neural network" is capable of processing input and providing corresponding output. It typically comprises an input layer, an output layer, and one or more hidden layers between the input and output layers. The layers in a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the inputs to the neural network model, while the output layer's output serves as the final output of the neural network model. Each layer of a neural network model includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the layer above. In this paper, the terms "neural network," "model," "network," and "neural network model" are used interchangeably.

[0017] As discussed above, super-resolution methods have been widely applied to various aspects of people's lives. Traditional super-resolution methods can be technically divided into three categories: (1) interpolation-based methods; (2) reconstruction-based methods; and (3) learning-based methods.

[0018] With the rapid development of artificial intelligence technology, learning-based methods have become the most popular super-resolution algorithms. This method uses training sample pairs (high-resolution images and corresponding low-resolution images) to calculate prior knowledge between high-resolution and low-resolution images and establish a mapping model between them.

[0019] Traditional learning-based super-resolution methods typically rely on pairs of high-resolution and low-resolution images. However, such high / low-resolution image pairs do not actually exist in real-world scenes. Therefore, some traditional super-resolution algorithms construct high / low-resolution image pairs by artificially synthesizing corresponding high-resolution or low-resolution images. These image pairs are then applied to machine learning models, enabling the models to generate high-resolution images from low-resolution images.

[0020] However, machine learning models trained on such image pairs often only perform well when processing low-resolution images obtained using the same downsampling method. For example, if the sample pairs used to train the model are low-resolution images obtained through interpolation, the model can often only obtain high-quality high-resolution images when processing similarly interpolated low-resolution images.

[0021] In most cases, people are unaware of how the low-resolution images to be processed were generated. In this context, super-resolution is also known as blind super-resolution. Because the downsampling methods used on artificially constructed image pairs are often different from those used on the low-resolution images to be processed, the trained machine learning models may struggle to achieve high-quality image super-resolution.

[0022] According to an implementation of this disclosure, a scheme for super-resolution image reconstruction is proposed. In this scheme, an invertible neural network is trained using an input image with a first resolution. The invertible neural network is configured to generate an intermediate image with a second resolution and first high-frequency information based on the input image, wherein the second resolution is lower than the first resolution. Subsequently, the inverse network of the trained invertible neural network is used to generate an output image with a third resolution based on the input image and the second high-frequency information following a predetermined distribution, wherein the third resolution is higher than the first resolution. Specifically, during the training of the invertible neural network, the low-resolution input image is converted into an even lower-resolution image so that the forward process of the invertible neural network can simulate a downsampling method for the input image location. Subsequently, the inverse network of the invertible neural network can be used to process the low-resolution input image to obtain a high-resolution output image.

[0023] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0024] Figure 1A A block diagram of a computing device 100 capable of implementing multiple implementations of the present disclosure is shown. It should be understood that... Figure 1A The computing device 100 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the implementation described in this disclosure. Figure 1A As shown, computing device 100 includes computing device 100 in the form of general computing device. Components of computing device 100 may include, but are not limited to, one or more processors or processing units 110, memory 120, storage device 130, one or more communication units 140, one or more input devices 150, and one or more output devices 160.

[0025] In some implementations, computing device 100 can be implemented as various user terminals or service terminals. Service terminals can be servers, large computing devices, etc., provided by various service providers. User terminals can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, sites, units, devices, multimedia computers, multimedia tablets, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that computing device 100 can support any type of user-facing interface (such as "wearable" circuitry).

[0026] Processing unit 110 can be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 120. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 100. Processing unit 110 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0027] Computing device 100 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 100, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 120 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof.

[0028] Storage device 130 may be a removable or non-removable medium and may include machine-readable media, such as memory, flash drives, disks, or any other media capable of storing information and / or data and accessible within computing device 100. Computing device 100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 1A As shown, disk drives for reading from or writing to removable, non-volatile disks and optical disc drives for reading from or writing to removable, non-volatile optical discs can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces.

[0029] The communication unit 140 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 100 can be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 100 can operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or another general network node.

[0030] Input device 150 can be one or more various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 160 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 100 can also communicate as needed with one or more external devices (not shown) via communication unit 140. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with computing device 100, or with any device that enables computing device 100 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0031] In some implementations, in addition to being integrated into a single device, some or all of the components of computing device 100 may be configured in the form of a cloud computing architecture. In a cloud computing architecture, these components can be remotely deployed and can work together to achieve the functionality described herein. In some implementations, cloud computing provides computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various implementations, cloud computing provides services over a wide area network (WAN), such as the Internet, using appropriate protocols. For example, cloud computing providers offer applications over a WAN, and these applications can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated at remote data center locations or they may be distributed. Cloud computing infrastructure can provide services through shared data centers, even if they appear as a single access point for users. Therefore, the components and functionality described herein can be provided from service providers at remote locations using a cloud computing architecture. Alternatively, they may be provided from conventional servers, or they may be installed directly or otherwise on client devices.

[0032] The computing device 100 can be used to implement image super-resolution reconstruction according to various implementations of the present disclosure. The memory 120 may include an image processing module 122 having one or more program instructions that can be accessed and executed by the processing unit 110 to implement the functions of the various implementations described herein.

[0033] During super-resolution image reconstruction, computing device 100 can receive input image 170 via input device 150. In some implementations, input image 170 may be, for example, an image with a first resolution. Input image 170 can be input into image processing module 122 in memory 120. Image processing module 122 can use input image 170 to train a reversible neural network such that the forward network of the reversible neural network can simulate the downsampling method of input image 170. The reversible neural network can convert input image 170 into an intermediate image with a lower resolution. Subsequently, the inverse network of the reversible neural network is used to process input image 170 to obtain an output image with a higher resolution, thereby achieving super-resolution image reconstruction. Output image 180 can be output via output device 160.

[0034] Figure 1B A schematic diagram illustrating the working principle of the image processing module 122 implemented according to this disclosure is shown. Figure 1B As shown, the image processing module 122 can use an input image 170 with a first resolution to train a reversible neural network 190 (represented as "..."). The reversible neural network 190 is configured to generate an intermediate image 192 with a lower second resolution and first high-frequency information 194 based on the input image 170. Further, after training the reversible neural network 190 using the input image 170, the input image 170 and the second high-frequency information 196 following a predetermined distribution are input into the inverse network 198 of the reversible neural network 190 (referred to as "..."). The method generates an output image 180 with a higher third resolution. The "predetermined distribution" mentioned herein may include, but is not limited to, a Gaussian distribution, a uniform distribution, etc., which can be specified during the training of the invertible neural network.

[0035] Invertible neural networks (INNs) are a popular network architecture in generative models that allow specifying mapping relationships. and its inverse mapping relationship An INN typically consists of at least one reversible block. For the l-th block, the input... Divided along the channel axis into and And undergoes an affine transformation:

[0036] (1)

[0037] (2)

[0038] The corresponding output is Given the output, its inverse transform can be calculated as follows:

[0039] (3)

[0040] (4)

[0041] Where φ, ρ, and η can be arbitrary functions. This indicates the OR operation.

[0042] When applying an INN to an image downsampling task, based on a high-resolution input image x, the INN can output not only a downsized low-resolution image y, but also high-frequency information z following a specific distribution, which may manifest as high-frequency noise independent of the image's semantics. This allows the inverse network of the INN to reconstruct a high-quality high-resolution image x from the low-resolution image y and the high-frequency information z. That is, it is necessary to maintain the high-frequency information z lost during image downsampling to make the image scaling process reversible, and the entire image scaling process can utilize mapping relationships. and To express.

[0043] However, image super-resolution reconstruction often requires upscaling any low-resolution image. Therefore, the high-frequency information z corresponding to the input low-resolution image is usually absent. The inventors noted that, according to the Nyquist-Shannon sampling theorem, the information lost during image downscaling corresponds to high-frequency details. Assuming a set of high-resolution images corresponding to the same low-resolution image contains different high-frequency details, these details can typically exhibit some variability and randomness. Therefore, z can be represented as a random variable, its distribution being represented by INN (i.e.,...). The output z is determined by the distribution. Specifically, an INN can be trained to satisfy a specified distribution. In this way, during image super-resolution reconstruction, a high-resolution image can be reconstructed based on a low-resolution image and any sample from the specified distribution.

[0044] Figure 2A A schematic block diagram of a reversible neural network 191 implemented according to this disclosure is shown. It should be understood that, as Figure 2A The structure of the reversible neural network 191 shown is merely exemplary and is not intended to limit the scope of this disclosure. Implementations of this disclosure are also applicable to reversible neural networks with different structures.

[0045] like Figure 2A As shown, the reversible neural network 191 can be composed of one or more downsampling modules 210 connected in series. For simplification purposes, in Figure 2AThe diagram shows a downsampling module 210. The image downsampling ratio supported by the reversible neural network 191 can be determined by the image downsampling ratio supported by each downsampling module 210 and the number of downsampling modules 210 included. For example, assuming each downsampling module 210 supports downsampling the image by a factor of 2 and the reversible neural network 191 includes 2 downsampling modules 210, then the reversible neural network 191 supports downsampling the image by a factor of 4.

[0046] like Figure 2A As shown, for example, the downsampling module 210 may include a transformation module 230 and one or more INN units 220-1, 220-2...220-M (collectively or individually referred to as "INN unit 220", where M≥1).

[0047] Transform module 230 can decompose the input image 170 into low-frequency components 242 and high-frequency components 241, where the low-frequency components 242 represent the semantics of the input image 170, and the high-frequency components 241 are related to the semantics. In some implementations, transform module 230 can be implemented using a 1×1 reversible convolution block. Alternatively, transform module 230 can also be implemented as a wavelet transform module, such as a Haar transform module. For example, when transform module 230 is implemented as a Haar transform module, downsampling module 210 can support downsampling the image by a factor of 2. Specifically, the Haar transform module can convert an input image or a set of feature maps with length H, width W, and number of channels C into an output tensor. The first C-slice in the output tensor can be approximated as a low-pass representation equivalent to bilinear interpolation downsampling. The remaining three sets of C-slices contain residual components in the vertical, horizontal, and diagonal directions, respectively. These residual components are based on high-frequency information in the input image 170. Alternatively, the transform module 230 can be implemented as any known or future-developed transform module capable of decomposing the input image 170 into low-frequency and high-frequency components. It should be understood that the implementation of the transform module 230 can differ depending on the image downscaling ratio supported by the downsampling module 210. In this way, low-frequency information 242 and high-frequency information 241 can be fed into the subsequent INN unit 220-1.

[0048] As described above, the structure of each INN unit 220 should be reversible, thereby ensuring the reversibility of the network structure of the neural network 191. The INN unit 220 is used to extract corresponding features from the low-frequency and high-frequency components of the input, and to convert the high-frequency components related to image semantics into high-frequency information that follows a predetermined distribution and is independent of image semantics.

[0049] Figure 2BA schematic diagram of an example INN unit 220 according to an implementation of this disclosure is shown. Here, it is assumed that the low-frequency and high-frequency components input to the INN unit 220 are represented as follows: and .like Figure 2B As shown, it can be directed to low-frequency components. Apply the affine transformation as shown in formula (1) above, and to the high-frequency components Apply the affine transformation as shown in formula (2) above. Figure 2B The transformation functions φ, η, and ρ shown can be arbitrary functions. It should be understood that, as... Figure 2B The INN unit 220 shown is for illustrative purposes only and is not intended to limit the scope of this disclosure. Implementations of this disclosure are also applicable to INN units with other different structures. Examples of INN units include, but are not limited to, reversible convolutional blocks, reversible residual network units, reversible generative network units, deep reversible network units, and so on.

[0050] The training process of the invertible neural network will be described in further detail below. As can be seen from the above description, the training objective of the model is to determine the high-resolution image x, the low-resolution image y, and a specified distribution. Mapping relationship between .

[0051] The inventors observed that when a high-resolution image is converted to a low-resolution image using an unknown downsampling method, the low-resolution image will have a similar pixel distribution to the corresponding image patch in the high-resolution image. Based on this, if the intermediate image 192 generated by the reversible neural network 190 has a similar pixel distribution to the corresponding image patch in the input image 170, it indicates that the reversible neural network 190 can effectively simulate the unknown downsampling method of the input image 170.

[0052] Based on this objective, in some implementations, the objective function (also known as the "first objective function") of the reversible neural network 190 can be determined based on the difference between the pixel distribution in the intermediate image 192 and the pixel distribution in the image patch with the second resolution of the input image 170.

[0053] In some implementations, a discriminator can be used to determine the pixel distribution differences between the intermediate image 192 and the corresponding image blocks in the input image 170. Figure 3 A schematic diagram 300 of a trained reversible neural network 190 implemented according to the present disclosure is shown.

[0054] like Figure 3As shown, during training, the discriminator 330 can be used to distinguish the pixel distribution differences between the intermediate image 192 generated by the reversible neural network 190 and the corresponding image patch 325 in the input image 170. In some implementations, the image patch 325 may be an image patch randomly selected from the input image 170 with the same pixel size as the intermediate image 192.

[0055] In some implementations, a discriminator 330 of a trained generative adversarial network (GAN) can be used to distinguish whether pixels in the intermediate image 192 come from the intermediate image 192 or from image patch 325. The loss function of the corresponding GAN can be determined as the first objective function of the invertible neural network 190, for example, it can be represented as the objective function. It should be understood that any appropriate GAN loss function can be used, such as JS divergence.

[0056] Additionally or alternatively, in some implementations, when training the reversible neural network 190, the first high-frequency information 194 generated by the reversible neural network 190 based on the input image 170 should also satisfy a predetermined distribution. Specifically, the objective function (also called the "second objective function") of the reversible neural network 190 can be determined based on the difference between the distribution of the first high-frequency information 194 and the predetermined distribution; for example, it can be expressed as an objective function. As discussed above, the "predetermined distribution" mentioned herein may include, but is not limited to, Gaussian distribution, uniform distribution, etc.

[0057] Additionally or alternatively, in some implementations, the objective function (also referred to as the "third objective function") used to train the invertible neural network 190 can be determined based on the difference between the input image 170 and the reconstructed image 315 generated by the inverse network 310 of the invertible neural network 190 based on the intermediate image 192. Specifically, the intermediate image 192 and high-frequency information obtained from sampling from a predetermined distribution can be input into the inverse network 310 to obtain the reconstructed image. The structure of the inverse network 310 will be referenced below. Figure 4A Detailed description.

[0058] In some implementations, the third objective function can be determined based on the L1 or L2 distance between the input image 170 and the reconstructed image 315; for example, it can be represented as an objective function. .

[0059] Additional or alternative locations, in some implementations, such as Figure 3As shown, a trained discriminator 320 can also be used to distinguish between the input image 170 and the reconstructed image 315 to determine the third objective function. For example, the discriminator 320 can be a discriminator in a trained generative adversarial network (GAN), and correspondingly, the third objective function can be set as the loss function of the GAN, for example, it can be expressed as... .

[0060] In some implementations, to accelerate the convergence of the invertible neural network 190 and ensure the semantic accuracy of the generated intermediate image 192, an objective function (also called a "fourth objective function") for training the invertible neural network 190 can be determined based on a reference image. For example, a reference image with a second resolution that semantically corresponds to the input image 170 can be obtained as training data for training the model. In some implementations, the reference image can be generated based on the input image 170. For example, interpolation or any known or planned method can be used to generate a low-resolution reference image semantically corresponding to the input image 170.

[0061] In some implementations, the fourth objective function used for training can be determined based on the difference between the reference image and the intermediate image 192 generated by the invertible neural network. In some implementations, the fourth objective function can be determined based on the L1 or L2 distance between the intermediate image 192 and the reference image; for example, it can be represented as an objective function. .

[0062] In some implementations, the overall objective function for training the invertible neural network 190 can be generated based on a combination of the first, second, and third objective functions. For example, the overall objective function can be expressed as:

[0063] L total = λ 1· + λ 2. + λ 3· + λ 4. + λ 5. (5)

[0064] in, λ 1. λ 2. λ 3. λ 4 and λ 5 is a coefficient used to balance the different loss terms. This is achieved by making the overall objective function... L total Minimization allows us to determine the parameters of the invertible neural network 190.

[0065] like Figure 1B As shown, after training of the reversible neural network 190 is completed, the input image 170 and the second high-frequency information 196 following a predetermined distribution can be input into the inverse network 198 of the trained reversible neural network 190 to obtain a high-resolution output image 180. Figure 4A It shows Figure 2A A schematic block diagram of the inverse network 198 of the reversible neural network 191 shown. (See diagram below.) Figure 4A As shown, the inverse network 198 can be formed by connecting one or more upsampling modules 410 in series. For simplification purposes, in Figure 4A An upsampling module 410 is shown. The image magnification ratio supported by the inverse network 198 can be determined by the image magnification ratio supported by each upsampling module 410 and the number of upsampling modules 410 included. For example, assuming that each upsampling module 410 supports magnifying the image by 2 times and the inverse network 198 includes 2 upsampling modules 410, then the inverse network 198 supports magnifying the image by 4 times.

[0066] like Figure 4A As shown, for example, the upsampling module 410 may include a transformation module 430 and one or more INN units 420-1, 420-2...420-M (collectively or individually referred to as "INN unit 420", where M≥1). The structure of the INN unit 420 is similar to that shown below. Figure 2B The structure of the INN unit 220 shown is inverse, for example, as Figure 4B As shown. Taking INN unit 420-M as an example, here we assume that the low-resolution input image 170 input to INN unit 420-M is represented as... And the high-frequency information 175, which follows a predetermined distribution, is represented as .like Figure 4B As shown, it can be directed to Apply the inverse transformation of the affine transformation as shown in formula (3) above, and to Apply the inverse transformation of the affine transformation as shown in formula (4) above. Figure 4B The transformation functions φ, η, and ρ shown can be arbitrary functions. It should be understood that, as... Figure 4B The INN unit 420 shown is for illustrative purposes only and is not intended to limit the scope of this disclosure. Implementations of this disclosure are also applicable to INN units with other different structures. Examples of INN units include, but are not limited to, reversible convolutional blocks, reversible residual network units, reversible generative network units, deep reversible network units, and so on.

[0067] like Figure 4AAs shown, one or more INN units 420 can convert a low-resolution input image 170 and second high-frequency information 196 following a predetermined distribution into high-frequency components 441 and low-frequency components 442 to be merged. (This is in contrast to...) Figure 2A Conversely to the transform module 240 shown, transform module 430 can combine high-frequency components 441 and low-frequency components 442 into an output image 180 with high resolution. In some implementations, when transform module 230 is implemented as a 1×1 deconvolution block, transform module 430 can be implemented as a 1×1 reversible convolution block. Alternatively, when transform module 230 is implemented as a wavelet transform module, transform module 430 can be implemented as an inverse wavelet transform module. For example, when transform module 230 is implemented as a Haar transform module, transform module 430 can be implemented as an inverse Haar transform module. Alternatively, transform module 430 can also be implemented as any known or future-developed transform module capable of combining low-frequency and high-frequency components into an image.

[0068] Embodiments of this disclosure utilize a reversible neural network to simulate the sampling process of an input image and use the inverse network of this reversible neural network to generate a high-resolution output image. Based on this approach, the implementation of this disclosure does not rely on paired image training data and can more accurately simulate the actual sampling process of the input image, thereby generating a high-quality, high-resolution output image.

[0069] Figure 5 A flowchart of a method 500 for image super-resolution reconstruction according to some implementations of the present disclosure is shown. Method 500 can be implemented by a computing device 100, for example, it can be implemented at an image processing module 122 in the memory 120 of the computing device 100. At 502, the computing device 100 acquires an input image having a first resolution. At 504, the computing device 100 trains a reversible neural network using the input image, the reversible neural network being configured to generate an intermediate image with a second resolution and first high-frequency information based on the input image, the second resolution being lower than the first resolution. At 506, the computing device 100 uses the inverse network of the trained reversible neural network to generate an output image with a third resolution based on the input image and second high-frequency information following a predetermined distribution, the third resolution being higher than the first resolution.

[0070] The following are some example implementations of this disclosure.

[0071] In a first aspect, this disclosure provides a computer-implemented method. The method includes: acquiring an input image having a first resolution; training a reversible neural network using the input image, the reversible neural network being configured to generate an intermediate image having a second resolution and first high-frequency information based on the input image, the second resolution being lower than the first resolution; and generating an output image having a third resolution, higher than the first resolution, based on the input image and the second high-frequency information following a predetermined distribution using an inverse network of the reversible neural network.

[0072] In some implementations, training a reversible neural network using an input image includes: determining multiple objective functions based on the input image and intermediate images; determining a total objective function for training the reversible neural network by combining at least a portion of the multiple objective functions; and determining the network parameters of the reversible neural network by minimizing the total objective function.

[0073] In some implementations, determining multiple objective functions includes: determining a first objective function based on the difference between the pixel distribution in the intermediate image and the pixel distribution in an image patch of the input image with a second resolution.

[0074] In some implementations, determining the first objective function includes: using a discriminator to distinguish whether the pixels of the intermediate image come from the intermediate image or from an image patch; and determining the first objective function based on the distinction.

[0075] In some implementations, determining multiple objective functions includes: determining a second objective function based on the difference between the distribution of a first high-frequency information and a predetermined distribution.

[0076] In some implementations, determining multiple objective functions includes: using the inverse network of a reversible neural network to generate a reconstructed image with a first resolution based on an intermediate image and third high-frequency information following a predetermined distribution; and determining a third objective function based on the difference between the input image and the reconstructed image.

[0077] In some implementations, determining multiple objective functions includes: obtaining a reference image that semantically corresponds to the input image and has a second resolution; and determining a fourth objective function based on the differences between the intermediate image and the reference image.

[0078] In some implementations, the reversible neural network includes a transformation module and at least one reversible network unit, wherein generating the output image includes: using at least one reversible network unit to generate low-frequency components and high-frequency components to be merged based on the input image and second high-frequency information, wherein the low-frequency components represent the semantics of the input image and the high-frequency components are semantically related; and using the transformation module to merge the low-frequency components and high-frequency components into the output image.

[0079] In some implementations, the transform module includes any of the following: a reversible convolutional block; and a wavelet transform module.

[0080] In a second aspect, this disclosure provides a device for managing an image. The device includes: a processing unit; and a memory coupled to the processing unit and containing instructions stored thereon, the instructions causing the device to perform actions when executed by the processing unit, the actions including: acquiring an input image having a first resolution; training a reversible neural network using the input image, the reversible neural network being configured to generate an intermediate image having a second resolution and first high-frequency information based on the input image, the second resolution being lower than the first resolution; and generating an output image having a third resolution based on the input image and second high-frequency information following a predetermined distribution using an inverse network of the reversible neural network, the third resolution being higher than the first resolution.

[0081] In some implementations, training a reversible neural network using an input image includes: determining multiple objective functions based on the input image and intermediate images; determining a total objective function for training the reversible neural network by combining at least a portion of the multiple objective functions; and determining the network parameters of the reversible neural network by minimizing the total objective function.

[0082] In some implementations, determining multiple objective functions includes: determining a first objective function based on the difference between the pixel distribution in the intermediate image and the pixel distribution in an image patch of the input image with a second resolution.

[0083] In some implementations, determining the first objective function includes: using a discriminator to distinguish whether the pixels of the intermediate image come from the intermediate image or from an image patch; and determining the first objective function based on the distinction.

[0084] In some implementations, determining multiple objective functions includes: determining a second objective function based on the difference between the distribution of a first high-frequency information and a predetermined distribution.

[0085] In some implementations, determining multiple objective functions includes: using the inverse network of a reversible neural network to generate a reconstructed image with a first resolution based on an intermediate image and third high-frequency information following a predetermined distribution; and determining a third objective function based on the difference between the input image and the reconstructed image.

[0086] In some implementations, determining multiple objective functions includes: obtaining a reference image that semantically corresponds to the input image and has a second resolution; and determining a fourth objective function based on the differences between the intermediate image and the reference image.

[0087] In some implementations, the reversible neural network includes a transformation module and at least one reversible network unit, wherein generating the output image includes: using at least one reversible network unit to generate low-frequency components and high-frequency components to be merged based on the input image and second high-frequency information, wherein the low-frequency components represent the semantics of the input image and the high-frequency components are semantically related; and using the transformation module to merge the low-frequency components and high-frequency components into the output image.

[0088] In some implementations, the transform module includes any of the following: a reversible convolutional block; and a wavelet transform module.

[0089] In a third aspect, a computer program product is provided. The computer program product is tangibly stored in a non-transient computer storage medium and includes machine-executable instructions that, when executed by a device, cause the device to perform actions, including: acquiring an input image having a first resolution; training a reversible neural network using the input image, the reversible neural network being configured to generate an intermediate image having a second resolution and first high-frequency information based on the input image, the second resolution being lower than the first resolution; and generating an output image having a third resolution, higher than the first resolution, based on the input image and the second high-frequency information following a predetermined distribution using the inverse network of the reversible neural network.

[0090] In some implementations, training a reversible neural network using an input image includes: determining multiple objective functions based on the input image and intermediate images; determining a total objective function for training the reversible neural network by combining at least a portion of the multiple objective functions; and determining the network parameters of the reversible neural network by minimizing the total objective function.

[0091] In some implementations, determining multiple objective functions includes: determining a first objective function based on the difference between the pixel distribution in the intermediate image and the pixel distribution in an image patch of the input image with a second resolution.

[0092] In some implementations, determining the first objective function includes: using a discriminator to distinguish whether the pixels of the intermediate image come from the intermediate image or from an image patch; and determining the first objective function based on the distinction.

[0093] In some implementations, determining multiple objective functions includes: determining a second objective function based on the difference between the distribution of a first high-frequency information and a predetermined distribution.

[0094] In some implementations, determining multiple objective functions includes: using the inverse network of a reversible neural network to generate a reconstructed image with a first resolution based on an intermediate image and third high-frequency information following a predetermined distribution; and determining a third objective function based on the difference between the input image and the reconstructed image.

[0095] In some implementations, determining multiple objective functions includes: obtaining a reference image that semantically corresponds to the input image and has a second resolution; and determining a fourth objective function based on the differences between the intermediate image and the reference image.

[0096] In some implementations, the reversible neural network includes a transformation module and at least one reversible network unit, wherein generating the output image includes: using at least one reversible network unit to generate low-frequency components and high-frequency components to be merged based on the input image and second high-frequency information, wherein the low-frequency components represent the semantics of the input image and the high-frequency components are semantically related; and using the transformation module to merge the low-frequency components and high-frequency components into the output image.

[0097] In some implementations, the transform module includes any of the following: a reversible convolutional block; and a wavelet transform module.

[0098] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0099] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0100] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0101] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of a single implementation may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0102] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A computer-implemented method, comprising: Obtain an input image with a first resolution; A reversible neural network is trained using the input image, the reversible neural network being configured to generate an intermediate image with a second resolution and first high-frequency information based on the input image, the second resolution being lower than the first resolution; The training of the invertible neural network using the input image includes: Based on the input image and the intermediate image, multiple objective functions are determined, and a first objective function is determined by the difference between the pixel distribution in the intermediate image and the pixel distribution in the image patch with the second resolution of the input image; as well as Using the inverse network of the trained reversible neural network, an output image with a third resolution is generated based on the input image and second high-frequency information following a predetermined distribution, the third resolution being higher than the first resolution.

2. The method according to claim 1, wherein training the reversible neural network using the input image further comprises: By combining at least a portion of the multiple objective functions, a total objective function for training the invertible neural network is determined; as well as The network parameters of the reversible neural network are determined by minimizing the overall objective function.

3. The method according to claim 2, wherein determining the first objective function comprises: A discriminator is used to distinguish whether the pixels of the intermediate image come from the intermediate image itself or from the image patch; And based on the distinction, the first objective function is determined.

4. The method of claim 2, wherein determining the plurality of objective functions includes: The second objective function is determined based on the difference between the distribution of the first high-frequency information and the predetermined distribution.

5. The method of claim 2, wherein determining the plurality of objective functions comprises: Using the inverse network of the reversible neural network, a reconstructed image with a first resolution is generated based on the intermediate image and third high-frequency information that follows the predetermined distribution. as well as A third objective function is determined based on the difference between the input image and the reconstructed image.

6. The method of claim 2, wherein determining the plurality of objective functions comprises: Obtain a reference image that semantically corresponds to the input image and has the second resolution; as well as A fourth objective function is determined based on the differences between the intermediate image and the reference image.

7. The method according to claim 1, wherein the reversible neural network comprises a transformation module and at least one reversible network unit, wherein generating the output image comprises: Using the at least one reversible network unit, low-frequency components and high-frequency components to be merged are generated based on the input image and the second high-frequency information, wherein the low-frequency components represent the semantics of the input image and the high-frequency components are related to the semantics; as well as The transformation module is used to combine the low-frequency components and the high-frequency components into the output image.

8. The method of claim 7, wherein the transformation module comprises any one of the following: Reversible convolutional blocks; and Wavelet transform module.

9. An apparatus comprising: Processing unit; as well as A memory, coupled to the processing unit and containing instructions stored thereon, which, when executed by the processing unit, cause the device to perform actions, including: Obtain an input image with a first resolution; A reversible neural network is trained using the input image, the reversible neural network being configured to generate an intermediate image with a second resolution and first high-frequency information based on the input image, the second resolution being lower than the first resolution; The training of the invertible neural network using the input image includes: Based on the input image and the intermediate image, multiple objective functions are determined. A first objective function is determined by comparing the pixel distribution in the intermediate image with the pixel distribution in an image patch of the input image having the second resolution. Using the inverse network of the trained reversible neural network, an output image with a third resolution is generated based on the input image and second high-frequency information following a predetermined distribution, the third resolution being higher than the first resolution.

10. The apparatus of claim 9, wherein training the reversible neural network using the input image further comprises: By combining at least a portion of the multiple objective functions, a total objective function for training the invertible neural network is determined; as well as The network parameters of the reversible neural network are determined by minimizing the overall objective function.

11. The device of claim 10, wherein determining the first objective function comprises: A discriminator is used to distinguish whether the pixels of the intermediate image come from the intermediate image itself or from the image patch; as well as Based on the distinction, the first objective function is determined.

12. The device of claim 9, wherein determining the plurality of objective functions comprises: The second objective function is determined based on the difference between the distribution of the first high-frequency information and the predetermined distribution.

13. The device of claim 9, wherein determining the plurality of objective functions comprises: Using the inverse network of the reversible neural network, a reconstructed image with a first resolution is generated based on the intermediate image and third high-frequency information that follows the predetermined distribution. as well as A third objective function is determined based on the difference between the input image and the reconstructed image.

14. The device of claim 9, wherein determining the plurality of objective functions comprises: Obtain a reference image that semantically corresponds to the input image and has the second resolution; as well as A fourth objective function is determined based on the differences between the intermediate image and the reference image.

15. The apparatus of claim 9, wherein the reversible neural network comprises a transformation module and at least one reversible network unit, wherein generating the output image comprises: Using the at least one reversible network unit, low-frequency components and high-frequency components to be merged are generated based on the input image and the second high-frequency information, wherein the low-frequency components represent the semantics of the input image and the high-frequency components are related to the semantics; as well as The transformation module is used to combine the low-frequency components and the high-frequency components into the output image.

16. The device of claim 15, wherein the conversion module comprises any one of the following: Reversible convolutional blocks; and Wavelet transform module.

17. A computer program product tangibly stored in a non-transitory computer storage medium and comprising machine-executable instructions that, when executed by a device, cause the device to perform an action, the action comprising: Obtain an input image with a first resolution; A reversible neural network is trained using the input image, the reversible neural network being configured to generate an intermediate image with a second resolution and first high-frequency information based on the input image, the second resolution being lower than the first resolution; The training of the invertible neural network using the input image includes: Based on the input image and the intermediate image, multiple objective functions are determined, and a first objective function is determined by the difference between the pixel distribution in the intermediate image and the pixel distribution in the image patch with the second resolution of the input image; as well as Using the inverse network of the trained reversible neural network, an output image with a third resolution is generated based on the input image and second high-frequency information following a predetermined distribution, the third resolution being higher than the first resolution.

18. The computer program product of claim 17, wherein training the reversible neural network using the input image further comprises: By combining at least a portion of the multiple objective functions, a total objective function for training the invertible neural network is determined; as well as The network parameters of the reversible neural network are determined by minimizing the overall objective function.

Images