Image super-resolution reconstruction method, device and computer readable storage medium

Abstract

This invention provides an image super-resolution reconstruction method, device, and computer-readable storage medium, comprising: acquiring a real low-resolution image to be reconstructed; inputting the real low-resolution image to be reconstructed into a trained end-to-end reconstruction model to obtain a reconstructed high-resolution image; wherein the reconstruction model includes a first variational autoencoder (VAE1), a second variational autoencoder (VAE2), and a mapping network. Compared with the prior art, the reconstruction model in this invention, including the first variational autoencoder (VAE1), the second variational autoencoder (VAE2), and the mapping network, improves upon traditional generative adversarial networks. Using this reconstruction model can enhance the network's generalization ability, thereby increasing the resolution of the reconstructed image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically provides an image super-resolution reconstruction method, device, and computer-readable storage medium. Background Technology

[0002] With the advent of the big data era and the development of facial recognition technology, massive amounts of facial data have been accumulated. However, due to the influence of shooting equipment, transmission channels, and storage devices, facial images are inevitably affected by various noises, thus damaging facial clarity and affecting the performance of other downstream visual tasks. Currently, generative adversarial networks (GANs) have been widely used in tasks such as image denoising and reconstruction due to their excellent image generation performance. This method requires pairs of low-resolution and high-resolution data as supervised training networks. However, considering that it is difficult to obtain pairs of low-resolution and high-resolution data in reality, existing methods generally use downsampling algorithms to construct data pairs. Although downsampling algorithms can simulate low-resolution data, the method is too simplistic and cannot represent real low-resolution data. That is, simulated low-resolution data and real low-resolution data are still in two different spaces with a certain distance, which leads to the network trained on simulated low-resolution data not performing well on real low-resolution data. In addition, existing methods still start from the image perspective, training a network to complete image super-resolution reconstruction. The reconstruction effect of such methods is highly dependent on the training image data, thus resulting in poor generalization.

[0003] Accordingly, there is a need in the field for a new image super-resolution reconstruction scheme to solve the above problems. Summary of the Invention

[0004] To overcome the above-mentioned defects, the present invention is proposed to provide a solution or at least a partial solution to the problem of low resolution in images obtained from image resolution reconstruction.

[0005] In a first aspect, the present invention provides an image super-resolution reconstruction method, comprising:

[0006] Obtain the actual low-resolution image to be reconstructed;

[0007] The real low-resolution image to be reconstructed is input into a trained end-to-end reconstruction model to obtain a reconstructed high-resolution image; wherein the reconstruction model includes a first variational autoencoder VAE1, a second variational autoencoder VAE2, and a mapping network.

[0008] In one technical solution of the above-mentioned image super-resolution reconstruction method, the real low-resolution image to be reconstructed is input into a trained end-to-end reconstruction model to obtain a reconstructed high-resolution image, including:

[0009] The real low-resolution image to be reconstructed is input into the second variational autoencoder VAE2 to obtain the low-resolution latent space of the real low-resolution image to be reconstructed.

[0010] The low-resolution latent space of the real low-resolution image to be reconstructed is converted to a high-resolution latent space through the mapping network;

[0011] The high-definition latent space is processed by the first variational autoencoder VAE1 to obtain the reconstructed high-definition image.

[0012] In one technical solution of the above image super-resolution reconstruction method, the first variational autoencoder (VAE1) includes a first encoder and a first decoder, and the method includes training the first variational autoencoder (VAE1) based on at least the following steps:

[0013] Acquire training sample data, including high-definition training data;

[0014] The real high-definition images of the high-definition training data are encoded into the real high-definition latent space based on the first encoder;

[0015] The real high-definition latent space is decoded based on the first decoder to obtain a real high-definition reconstructed image;

[0016] A first total loss function is obtained based at least on the real high-definition image, the real high-definition reconstructed image, and the real high-definition latent space;

[0017] The first variational autoencoder VAE1 is trained based on the first total loss function.

[0018] In one technical solution of the above image super-resolution reconstruction method, obtaining the first total loss function includes:

[0019] Based on the real high-definition reconstructed image, a first reconstruction loss function is obtained from the real high-definition image;

[0020] Based on the real high-definition latent space, the first KL divergence loss function is obtained;

[0021] Based on the real high-definition reconstructed image, the real high-definition image, and the first variational autoencoder VAE1, the first adversarial loss function is obtained.

[0022] The sum of the first reconstruction loss function, the first KL divergence loss function, and the first adversarial loss function is the first total loss function.

[0023] In one technical solution of the above-mentioned image super-resolution reconstruction method, the second variational autoencoder (VAE2) includes a second encoder and a second decoder, and the method includes training the second variational autoencoder (VAE2) based on at least the following steps:

[0024] Acquire low-resolution training data, which includes simulated low-resolution images and real low-resolution images;

[0025] Acquire simulated low-resolution images and real low-resolution images;

[0026] Based on the second encoder, the simulated low-resolution image and the real low-resolution image are encoded respectively to obtain the simulated low-resolution latent space and the real low-resolution latent space.

[0027] The simulated low-resolution latent space and the real low-resolution latent space are decoded based on the second decoder to obtain the simulated low-resolution reconstructed image and the real low-resolution reconstructed image.

[0028] A second total loss function is obtained based at least on the simulated low-resolution latent space, the real low-resolution latent space, the simulated low-resolution reconstructed image, and the real low-resolution reconstructed image;

[0029] The second variational autoencoder VAE2 is trained based on the second total loss function.

[0030] In one technical solution of the above image super-resolution reconstruction method, obtaining the second total loss function includes:

[0031] Obtain the first sub-loss function, the second sub-loss function, and the fourth adversarial loss function. The sum of the first sub-loss function, the second sub-loss function, and the fourth adversarial loss function is the second total loss function.

[0032] The process of obtaining the first sub-loss function includes:

[0033] Based on the simulated low-resolution reconstructed image and the simulated low-resolution image, a second reconstruction loss function is obtained;

[0034] Based on the simulated low-clearance potential space, the second KL divergence loss function is obtained;

[0035] Based on the simulated low-resolution reconstructed image, the simulated low-resolution image, and the second variational autoencoder VAE2, a second adversarial loss function is obtained.

[0036] The sum of the second reconstruction loss function, the second KL divergence loss function, and the second adversarial loss function constitutes the first sub-loss function;

[0037] Obtaining the second sub-loss function includes:

[0038] Based on the real low-resolution reconstructed image, a third reconstruction loss function is obtained;

[0039] The third KL divergence loss function is obtained based on the real low-clearance potential space;

[0040] Based on the real low-resolution reconstructed image, the real low-resolution image, and the second variational autoencoder VAE2, a third adversarial loss function is obtained.

[0041] The sum of the third reconstruction loss function, the third KL divergence loss function, and the third adversarial loss function is the second sub-loss function;

[0042] Based on the simulated low-level latent space, the real low-level latent space, and the second variational autoencoder VAE2, the fourth adversarial loss function is obtained.

[0043] In one technical solution of the above image super-resolution reconstruction method, training the mapping network includes:

[0044] The analog low-resolution image is encoded into the analog low-resolution latent space based on the second encoder;

[0045] The simulated low-resolution latent space is converted to high-resolution latent space through a mapping network, and then decoded by the second variational autoencoder VAE2 to obtain the reconstructed high-resolution image;

[0046] The real high-definition image is encoded into the real high-definition latent space based on the second variational autoencoder VAE2.

[0047] The third total loss function is obtained based at least on the real high-definition latent space, the reconstructed high-definition image, the simulated low-definition image, the first variational autoencoder VAE1, and the second variational autoencoder VAE2.

[0048] The mapping network is trained based on the third total loss function.

[0049] In one technical solution of the above image super-resolution reconstruction method, obtaining the third total loss function includes:

[0050] Based on the high-definition latent space transformed by the mapping network and the real high-definition latent space, the latent space distance loss function is obtained;

[0051] Based on the reconstructed high-definition image and the real high-definition image, a fifth adversarial loss is obtained;

[0052] Based on the reconstructed high-definition image, the real high-definition image, and the activation function, the perceptual loss is obtained.

[0053] The sum of the latent space distance loss function, the fifth adversarial loss, and the perception loss constitutes the third total loss function.

[0054] In a second aspect, the present invention provides an electronic device including a processor and a storage device, the storage device being adapted to store a plurality of program codes, the program codes being adapted to be loaded and run by the processor to perform the image super-resolution reconstruction method described in any of the above-described technical solutions.

[0055] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the image super-resolution reconstruction method described in any of the above-described technical solutions.

[0056] The above-described technical solutions of the present invention have at least one or more of the following beneficial effects:

[0057] In implementing the technical solution of this invention, the present invention provides an image super-resolution reconstruction method, comprising: acquiring a real low-resolution image to be reconstructed; inputting the real low-resolution image to be reconstructed into a trained end-to-end reconstruction model to obtain a reconstructed high-resolution image; wherein the reconstruction model includes a first variational autoencoder (VAE1), a second variational autoencoder (VAE2), and a mapping network. Compared with the prior art, the beneficial effects of the image super-resolution reconstruction method provided by this invention are as follows: the reconstruction model in this scheme includes a first variational autoencoder (VAE1), a second variational autoencoder (VAE2), and a mapping network, which improves upon the traditional generative adversarial network. Using this reconstruction model can improve the network's generalization ability, thereby enhancing the resolution of the reconstructed image.

[0058] Furthermore, in this scheme, the real low-resolution image to be reconstructed is input into the second variational autoencoder (VAE2) to obtain the low-resolution latent space of the real low-resolution image to be reconstructed; the low-resolution latent space representation of the real low-resolution image to be reconstructed is converted into a real high-resolution latent space through a mapping network; and the real high-resolution latent space is processed by the first variational autoencoder (VAE1) to obtain the reconstructed high-resolution image. Operating in the latent space, since the latent space represents the core features of the image, has a stronger generalization ability compared to operating from the image space. Attached Figure Description

[0059] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Furthermore, similar numbers in the drawings are used to denote similar components, wherein:

[0060] Figure 1 This is a schematic diagram of the main steps of an image super-resolution reconstruction method according to an embodiment of the present invention;

[0061] Figure 2 This is a schematic diagram of the main steps for training a first variational autoencoder VAE1 according to an embodiment of the present invention.

[0062] Figure 3 This is a schematic diagram of the encoder structure according to an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of the decoder structure according to an embodiment of the present invention;

[0064] Figure 5 This is a schematic diagram of the structure of a mapping network according to an embodiment of the present invention;

[0065] Figure 6 This is a schematic diagram of the main steps in training a mapping network according to an embodiment of the present invention. Detailed Implementation

[0066] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0067] In the description of this invention, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0068] Here we will first explain some of the terms involved in this invention.

[0069] Low-resolution data refers to data with lower resolution and some noise (such as jagged edges, image compression, blurriness, and color distortion).

[0070] High-definition data refers to data with high resolution, clean images, and no noise.

[0071] Latent space: The core representation of the data is obtained through a relevant feature extraction network.

[0072] To address the problem of existing methods treating simulated low-resolution data and real low-resolution data as belonging to the same domain, resulting in unsatisfactory super-resolution performance, this invention treats them as data from different domains. A Visual Image Representation (VAE) is used to encode both types of data into a latent space. An adversarial loss is introduced into the latent space to bridge the gap between the two, ensuring that the simulated low-resolution data can adequately represent the real low-resolution data. Based on this, a mapping network is used to convert from the low-resolution latent space to the high-resolution latent space, thereby achieving image super-resolution.

[0073] Example 1

[0074] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of an image super-resolution reconstruction method according to an embodiment of the present invention. Figure 1 As shown, the image super-resolution reconstruction method in this embodiment of the invention mainly includes the following steps S1-S2.

[0075] Step S1: Obtain the real low-resolution image to be reconstructed;

[0076] Specifically, the low-resolution images to be reconstructed are images whose clarity is impaired by noise. For example, ultrasound images obtained through ultrasound imaging equipment are affected by various factors such as acquisition distance, ambient lighting, and compression distortion, as well as blurry, low-resolution, and low-quality facial images in applications such as video surveillance.

[0077] Step S2: Input the real low-resolution image to be reconstructed into the trained end-to-end reconstruction model to obtain the reconstructed high-resolution image; wherein, the reconstruction model includes a first variational autoencoder VAE1, a second variational autoencoder VAE2, and a mapping network.

[0078] In this embodiment, the reconstruction model includes a first variational autoencoder (VAE1), a second variational autoencoder (VAE2), and a mapping network. It improves upon the traditional generative adversarial network. Using this reconstruction model can enhance the network's generalization ability, thereby increasing the resolution of the reconstructed image.

[0079] In one embodiment, inputting the real low-resolution image to be reconstructed into a trained end-to-end reconstruction model to obtain a reconstructed high-resolution image includes: inputting the real low-resolution image to be reconstructed into a second variational autoencoder (VAE2) to obtain a low-resolution latent space of the real low-resolution image to be reconstructed; converting the low-resolution latent space of the real low-resolution image to be reconstructed into a high-resolution latent space through the mapping network; and processing the high-resolution latent space through a first variational autoencoder (VAE1) to obtain the reconstructed high-resolution image.

[0080] In this embodiment, from the perspective of latent space, the real low-resolution image to be reconstructed is encoded into a low-resolution latent space of the real low-resolution image to be reconstructed, the low-resolution latent space is converted to obtain a high-resolution latent space, and finally the high-resolution latent space is decoded to obtain the reconstructed high-resolution image. Since the latent space is the expression of the core features of the image, it has a stronger generalization ability compared to operating from the image space.

[0081] In one embodiment, the first variational autoencoder (VAE1) includes a first encoder and a first decoder, such as... Figure 2 As shown, the method includes training a first variational autoencoder (VAE1) based on at least the following steps: acquiring training sample data, the training sample data including high-resolution training data; and converting real high-resolution images I from the high-resolution training data into high-resolution images based on the first encoder. y Encoded into the true high-definition latent space Z y Based on the first decoder, the real high-definition latent space Z y Decoding is performed to obtain a true high-definition reconstructed image I′ y At least based on the real high-definition image I y The real high-definition reconstructed image I′ y And the aforementioned real high-definition latent space Z y Obtain the first total loss function; train the first variational autoencoder VAE1 based on the first total loss function.

[0082] In this embodiment, the latent space Z of a real high-definition image is obtained by training the first variational autoencoder VAE1. y To ensure the effectiveness of training the first variational autoencoder VAE1, a first total loss function is constructed, and the first variational autoencoder VAE1 is trained based on the first total loss function.

[0083] In the implementation of VAE1 and VAE2, convolutional neural networks (CNNs) of any level can be used, such as ResNet and VGG. While commonly used CNNs can also be used in VAEs, they were initially designed primarily for image classification tasks. CNN structures typically employ batch normalization layers to reduce overfitting and gradient vanishing during training. However, this normalization method of subtracting the mean and dividing by the variance essentially destroys the image's original style and color information. Image super-resolution aims to ensure that the network output image maintains consistency with the input image in terms of color, contrast, and brightness, changing only the resolution and detail information. Therefore, applying BN layers naturally introduces drawbacks, affecting the super-resolution results. Thus, this embodiment borrows from existing CNN structures, such as ResBlock, but replaces the batch normalization layers with instance normalization layers, designing a corresponding network structure.

[0084] Both the first encoder and the second encoder consist of six layers, such as Figure 3 The diagram shows a series of convolutional layers arranged sequentially: a first convolutional layer (conv), a second convolutional layer (conv), a third convolutional layer (conv), two ResBlock layers, and a fourth convolutional layer (conv). Each convolutional layer is followed by an instance normalization layer and a ReLU function. For example, for an input S*S*3 RGB image, the output dimension of the first encoder is...

[0085] Both the first decoder and the second decoder consist of seven layers, such as Figure 4 Specifically, this includes sequentially configured ResBlock, ConvTranspose+instance norm+ReLU, two ResBlock layers, two ConvTranspose+instancenorm+ReLU layers, and one Conv+Tanh layer. In one implementation, obtaining the first total loss function includes: based on the real high-resolution reconstructed image I′ y Real high-definition images y Obtain the first reconstruction loss function; based on the real high-definition latent space Z y Obtain the first KL divergence loss function; based on the real high-definition reconstructed image I′ y Real high-definition images y The first variational autoencoder (VAE1) is used to obtain the first adversarial loss function; the sum of the first reconstruction loss function, the first KL divergence loss function, and the first adversarial loss function is the first total loss function.

[0086] In this embodiment, to ensure the effective training of the first variational autoencoder (VAE1), three types of losses can be optimized based on the training process of the first variational autoencoder (VAE1). Among them, to maintain the true high-definition reconstructed image I′ after the decoder... y With real high-definition images I y To achieve pixel-level consistency, the reconstruction loss needs to be optimized; therefore, the first reconstruction loss function... The expression is:

[0087]

[0088] The first variational autoencoder (VAE1) treats the latent space distribution as a standard Gaussian prior with a mean of 0 and a variance of 1. Therefore, it is necessary to optimize the KL divergence loss from the latent space to the Gaussian prior distribution. The first KL divergence loss function... The expression is:

[0089]

[0090] In the formula, μ and σ are the mean and variance learned by the first variational autoencoder VAE1, respectively.

[0091] To determine the true high-resolution reconstructed image I′ y To ensure authenticity, a first discriminator is introduced. The goal of the first discriminator is to distinguish the real high-resolution reconstructed image I′ as much as possible. y With real high-definition images I y Therefore, it is necessary to use the first minimization criterion for binary classification cross-entropy loss L. D1 To optimize VAE1, L D1 The expression is as follows:

[0092] L D1 =-label*logP-(1-label)log(1-P)

[0093] In the formula, P represents the image that is judged to be a real high-definition image I. y The probability is given by label=1, which represents a true high-definition image, and label=0, which represents a true high-definition reconstructed image.

[0094] Conversely, VAE1 aims to reconstruct a more realistic image to deceive the first discriminator, and can employ a first adversarial loss L. G1 To optimize VAE1, L G1 The expression is as follows:

[0095] L G1 =-logP

[0096] The training method involves VAE1 and the first discriminator competing against each other to motivate VAE1 to learn better reconstruction results.

[0097] Therefore, the expression for the first total loss function is:

[0098] in,

[0099] In one embodiment, the second variational autoencoder (VAE2) includes a second encoder and a second decoder, and the method includes training the second variational autoencoder (VAE2) based on at least the following steps: acquiring low-resolution training data, the low-resolution training data including simulated low-resolution images I. x and real low-resolution images I r ; Acquire simulated low-resolution image I x and real low-resolution images I r Based on the second encoder, the simulated low-resolution image I is processed respectively. x Real low-resolution image I r Encoding is performed to obtain the simulated low-clearance latent space Z. xand real low-definition potential space Z r Based on the second decoder, the simulated low-clearance potential space Z is respectively... x Real Low-Definition Potential Space Z r Decoding is performed to obtain the simulated low-resolution reconstructed image I′. x True low-resolution reconstructed image I′ r At least based on the simulated low-clearance potential space Z x The aforementioned true low-definition potential space Z r The simulated low-resolution reconstructed image I′ x True low-resolution reconstructed image I′ r Obtain the second total loss function; train the second variational autoencoder VAE2 based on the second total loss function.

[0100] In this embodiment, the simulation of the low-clearance potential space Z is completed by training the second variational autoencoder VAE2. x and real low-definition potential space Z r To ensure the effectiveness of training the second variational autoencoder (VAE2), a second total loss function is constructed, and the second variational autoencoder (VAE2) is optimized based on the second total loss function.

[0101] In one implementation, obtaining the second total loss function includes:

[0102] Obtain the first sub-loss function, the second sub-loss function, and the fourth adversarial loss function. The sum of the first sub-loss function, the second sub-loss function, and the fourth adversarial loss function is the second total loss function.

[0103] The first sub-loss function is obtained by: based on the simulated low-resolution reconstructed image I′ x Simulated low-resolution image I x Obtain the second reconstruction loss function; based on the simulated low-clearance potential space Z x Obtain the second KL divergence loss function; based on the simulated low-resolution reconstructed image I′ x Simulated low-resolution image I x The second variational autoencoder (VAE2) is used to obtain the second adversarial loss function; the sum of the second reconstruction loss function, the second KL divergence loss function, and the second adversarial loss function is the first sub-loss function.

[0104] Obtaining the second sub-loss function includes: based on the real low-resolution reconstructed image I′ r Real low-resolution image I r Obtain the third reconstruction loss function; based on the true low-clearance potential space Z r Obtain the third KL divergence loss function; reconstruct the image I′ based on the real low-resolution image. r Real low-resolution image I rThe second variational autoencoder (VAE2) is used to obtain the third adversarial loss function; the sum of the third reconstruction loss function, the third KL divergence loss function, and the third adversarial loss function is the second sub-loss function.

[0105] Based on the simulated low-clearance potential space Z x The aforementioned true low-definition potential space Z r The second variational autoencoder (VAE2) is used to obtain the fourth adversarial loss function.

[0106] In this embodiment, to ensure the effective training of the second variational autoencoder (VAE2), the training of the second variational autoencoder (VAE2) based on simulated low-resolution images and real low-resolution images optimizes three types of losses. Among them, to maintain the quality of the simulated low-resolution reconstructed image I′ after the second decoder... x With simulated low-resolution image I x To achieve pixel-level consistency, the reconstruction loss needs to be optimized; therefore, a second reconstruction loss function for low-resolution images is simulated. The expression is:

[0107]

[0108] The objective of this invention is to narrow the gap between the latent spaces of real low-resolution images and simulated low-resolution images, which requires the latent space to be continuous. Therefore, the second variational autoencoder (VAE2) achieves latent space continuity during training by optimizing the KL divergence loss. Specifically, the VAE2 treats the latent space distribution as a standard Gaussian prior with a mean of 0 and a variance of 1. Therefore, it is necessary to optimize the KL divergence loss from the latent space to the Gaussian prior distribution, simulating the low-resolution second KL divergence loss function. The expression is:

[0109]

[0110] In the formula, μ and σ are the mean and variance learned by the second variational autoencoder VAE2, respectively.

[0111] To determine the true high-resolution reconstructed image I′ y To ensure authenticity, a second discriminant is introduced. The goal of the second discriminant is to distinguish the simulated low-resolution reconstructed image I′ as much as possible. x With simulated low-resolution image I x Therefore, it is necessary to use the second minimization criterion for binary cross-entropy loss. To optimize VAE2, The expression is as follows:

[0112]

[0113] In the formula, P represents the image being judged as a simulated low-resolution image I. xThe probability of the simulated low-resolution image is given by label=1, and the probability of the simulated low-resolution reconstructed image is given by label=0.

[0114] Conversely, VAE2 aims to reconstruct a more realistic image to deceive the second discriminator, and can employ a second adversarial loss. To optimize VAE2, The expression is as follows:

[0115]

[0116] Therefore, the first sub-loss function The expression is:

[0117]

[0118] in,

[0119] The training method involves VAE2 and a second discriminator competing against each other to motivate VAE2 to learn better reconstruction results.

[0120] Similarly, the second sub-loss function The expression is:

[0121]

[0122] In the formula, Represents the third reconstruction loss function, Represents the third KL divergence loss function, This represents the third adversarial loss function. This represents the third minimum criterion for binary classification cross-entropy loss. This indicates the loss in the third confrontation.

[0123] In addition, this step introduces a fourth discriminator to examine the simulated low-clearance potential space Z. x With real low-definition potential space Z r The gap. The discriminator aims to distinguish the simulated low-clearance latent space Z. x With real low-definition potential space Z r Therefore, it is necessary to minimize the standard fourth binary classification cross-entropy loss L. D4 :

[0124] L D4 =-label*logP-(1-label)log(1-P)

[0125] Where P represents the determination that the latent space is a simulated low-clearance latent space Z. x The probability of.

[0126] Conversely, VAE2 aims to further narrow the simulated low-clearance potential space Z. x With real low-definition potential space Z r The distance is used to deceive the fourth discriminator, with a loss of L. G4 :

[0127] L G4 =-logP

[0128] Through the adversarial training method involving VAE2 and the fourth discriminator, the true low-resolution latent space Z is made more accurate. r Closer to simulating low-clearance potential space Z x .

[0129] Therefore, the expression for the second total loss function is:

[0130] in,

[0131] Furthermore, the process of acquiring simulated low-resolution images includes processing existing real high-resolution images using five different algorithms to generate corresponding simulated low-resolution images. The five algorithms are: downsampling, Gaussian blur, Gaussian noise, JPG compression, and motion blur. Downsampling is a mandatory algorithm used to destroy image details and reduce image resolution; the other four algorithms are randomly selected from two and combined in different orders.

[0132] In one implementation, such as Figure 6 As shown, training the mapping network includes: using the second encoder to simulate low-resolution images I x Encoding to the simulated low-clear latent space Z x The simulated low-clearance potential space Z x After being converted to high-definition latent space Z by a mapping network x→y The reconstructed high-definition image I is obtained by decoding with the second variational autoencoder VAE2. x→y Based on the second variational autoencoder VAE2, real high-definition images I y Encoded to true high-definition latent space Z y At least based on the aforementioned real high-definition latent space Z y The reconstructed high-definition image I x→y Simulated low-resolution image I x The first variational autoencoder (VAE1) and the second variational autoencoder (VAE2) obtain a third total loss function; the mapping network is trained based on the third total loss function.

[0133] In this embodiment, the parameters of the second encoder of VAE2 and the first decoder of VAE1 are frozen, based on the simulated low-resolution image I. x and corresponding real high-definition images Iy The paired data (I) constituted x I y Train the mapping network, the structure of which is as follows: Figure 5 As shown, the network consists of four Conv+instance norm+ReLU modules, six ResBlock modules, three Conv+instance norm+ReLU modules, and one Conv module. To ensure the effectiveness of the mapping network training, a third total loss function is constructed based on the training process, and the mapping network is optimized based on this third total loss function.

[0134] In one implementation, obtaining the third total loss function includes: based on the high-definition latent space Z after mapping network transformation. x→y The aforementioned real high-definition latent space Z y Obtain the latent space distance loss function; based on the reconstructed high-resolution image I x→y The real high-definition image I y Obtain the fifth adversarial loss; based on the reconstructed high-resolution image I x→y The real high-definition image I y Activation function Obtain the perception loss; the sum of the latent space distance loss function, the fifth adversarial loss, and the perception loss is the third total loss function.

[0135] In this embodiment, based on the training process of the mapping network, the loss can be divided into latent spatial distance loss, fifth adversarial loss, and perceptual loss. Among them, latent spatial distance loss is used to measure the Z-axis after transformation by the mapping network. x→y With real high-definition latent space Z y Distance between:

[0136]

[0137] To determine the reconstructed high-resolution image I x→y To ensure authenticity, the fifth discriminator is introduced to distinguish real high-definition images as much as possible. y Compared with the reconstructed high-resolution image I x→y Therefore, the fifth minimization criterion for binary cross-entropy loss needs to be adopted. To optimize the mapping network, The expression is as follows:

[0138]

[0139] In the formula, P represents the image that is judged to be a real high-definition image I. y The probability, label=1 represents the real high-resolution image I ylabel=0 indicates the reconstructed high-resolution image I x→y .

[0140] Conversely, the mapping network aims to reconstruct a more realistic image to deceive the fifth discriminator, which can employ a fifth adversarial loss. To optimize the mapping network, The expression is as follows:

[0141]

[0142] Perceptual loss is used to measure the consistency between the super-resolution image and the real high-resolution image from both visual and stylistic perspectives. In computation, perceptual loss is defined as the ratio of I... x→y with I y Simultaneously fed into a VGG network with fixed parameters, different layer characteristics L perceptual After activation function The subsequent loss L perceptual :

[0143]

[0144] Example 2

[0145] Unlike Embodiment 1, the reconstruction model includes a first autoencoder AE1, a second autoencoder AE2, and a mapping network. The first autoencoder AE1 includes a third encoder and a third decoder, and the second autoencoder AE2 includes a fourth encoder and a fourth decoder. The training methods for the first autoencoder AE1, the second autoencoder AE2, and the mapping network are the same as in Embodiment 1.

[0146] To ensure the effectiveness of training the first autoencoder AE1, a fourth total loss function is constructed, and the first autoencoder AE1 is trained based on this fourth total loss function. Specifically, two losses can be optimized according to the training process of the first autoencoder AE1, among which, to maintain the true high-definition reconstructed image I′ after decoding... y With real high-definition images I y To achieve pixel-level consistency, the reconstruction loss needs to be optimized; therefore, the sixth reconstruction loss function... The expression is:

[0147]

[0148] To determine the true high-resolution reconstructed image I′ y To ensure authenticity, a sixth discriminator is introduced. The goal of the sixth discriminator is to distinguish the real high-resolution reconstructed image I′ as much as possible. y With real high-definition images I y Therefore, the sixth minimization criterion for binary cross-entropy loss L needs to be adopted. D6To optimize the first autoencoder AE1, L D6 The expression is as follows:

[0149] L D6 =-label*logP-(1-label)log(1-P)

[0150] In the formula, P represents the image that is judged to be a real high-definition image I. y The probability is given by label=1, which represents a true high-definition image, and label=0, which represents a true high-definition reconstructed image.

[0151] Conversely, the first autoencoder AE1 aims to reconstruct a more realistic image to deceive the sixth discriminator, which can be achieved using the sixth adversarial loss L. G6 To optimize the first autoencoder AE1, L G6 The expression is as follows:

[0152] L G6 =-logP

[0153] The first autoencoder AE1 is trained by engaging in a competition between the first autoencoder AE1 and the sixth discriminator to improve its reconstruction performance.

[0154] Therefore, the expression for the fourth total loss function is:

[0155] in,

[0156] To ensure the effectiveness of the training of the second autoencoder AE2, a fifth total loss function is constructed, and the second autoencoder AE2 is trained based on this fifth total loss function. Specifically, three types of losses can be optimized according to the training process of the second autoencoder AE2. Two types of losses are optimized for training the second autoencoder AE2 based on both simulated low-resolution images and real low-resolution images:

[0157] To preserve the low-resolution reconstructed image I′ after being simulated by the third decoder x With simulated low-resolution image I x To achieve pixel-level consistency, the reconstruction loss needs to be optimized; therefore, the seventh reconstruction loss function for low-resolution images is simulated. The expression is:

[0158]

[0159] To determine the true high-resolution reconstructed image I′ y To ensure authenticity, a seventh discriminator is introduced. The goal of the seventh discriminator is to distinguish the simulated low-resolution reconstructed image I′ as much as possible. x With simulated low-resolution image I x Therefore, the seventh minimization criterion for binary cross-entropy loss needs to be adopted. To optimize the second automatic encoder AE2, The expression is as follows:

[0160]

[0161] In the formula, P represents the image being judged as a simulated low-resolution image I. x The probability of the simulated low-resolution image is given by label=1, and the probability of the simulated low-resolution reconstructed image is given by label=0.

[0162] Conversely, the second autoencoder AE2 aims to reconstruct a more realistic image to deceive the seventh discriminator, which can employ a seventh adversarial loss. To optimize the second automatic encoder AE2, The expression is as follows:

[0163]

[0164] Therefore, the third sub-loss function The expression is:

[0165]

[0166] in,

[0167] The VAE2 is trained by pitting the second autoencoder AE2 against the seventh discriminator to improve its reconstruction performance.

[0168] Similarly, the fourth sub-loss function The expression is:

[0169]

[0170] In the formula, This represents the eighth reconstruction loss function. This represents the eighth adversarial loss function. This represents the eighth minimum criterion for binary classification cross-entropy loss. This indicates the eighth type of resistance loss.

[0171] In addition, this step introduces a ninth discriminator to examine the simulated low-clearance potential space Z. x With real low-definition potential space Z r The gap. The goal of the ninth discriminator is to distinguish the simulated low-clearance latent space Z. x With real low-definition potential space Z r Therefore, the ninth criterion for minimizing the binary cross-entropy loss L is required. D9 :

[0172] LD9 =-label*logP-(1-label)log(1-P)

[0173] Where P represents the determination that the latent space is a simulated low-clearance latent space Z. x The probability of.

[0174] Conversely, the second autoencoder AE2 aims to further narrow the analog low-clearance potential space Z. x With real low-definition potential space Z r The distance is used to deceive the ninth discriminator, with a loss of L. G9 :

[0175] L G9 =-logP

[0176] The training method, which involves adversarial interaction between the second autoencoder AE2 and the ninth discriminator, enables the real low-resolution latent space Z to be optimized. r Closer to simulating low-clearance potential space Z x .

[0177] Therefore, the expression for the fifth total loss function is:

[0178] in

[0179] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of the present invention.

[0180] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0181] Furthermore, the present invention also provides an electronic device. In one embodiment of the electronic device according to the present invention, the electronic device includes a processor and a storage device. The storage device can be configured to store a program for performing the image super-resolution reconstruction method of the above-described method embodiments. The processor can be configured to execute the program in the storage device. The program includes, but is not limited to, performing the image super-resolution reconstruction method of the above-described method embodiments. For details not disclosed, please refer to the method section of the embodiments of the present invention. The electronic device can be an electronic device comprising various electronic devices.

[0182] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program for performing the image super-resolution reconstruction method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described image super-resolution reconstruction method. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0183] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, hardware, or a combination of software and hardware. Therefore, the number of modules shown in the figures is merely illustrative.

[0184] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of the present invention; therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.

[0185] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. An image super-resolution reconstruction method, characterized in that, The method comprises: obtaining a real low-resolution image to be reconstructed; inputting the real low-resolution image to be reconstructed into a trained end-to-end reconstruction model to obtain a reconstructed high-definition image; wherein the reconstruction model comprises a first variational autoencoder VAE1, a second variational autoencoder VAE2, and a mapping network; the second variational autoencoder VAE2 comprises a second encoder and a second decoder, and the method comprises training the second variational autoencoder VAE2 based on at least the following steps: acquiring low-quality training data, the low-quality training data including simulated low-quality images and real low-quality images ; Acquiring simulated low definition images and real low definition images ; Based on the second encoder, the simulated low-resolution images are processed respectively. Real low-resolution images Encoding is performed to obtain a simulated low-clearance potential space. and real low-definition potential space ; decoding, respectively, based on a second decoder, the simulated low definition latent space , the real low definition latent space to obtain the simulated low definition reconstructed image , the real low definition reconstructed image ; based on the simulated low-clearance latent space , the real low-clearance latent space , the simulated low-clearance reconstructed image , the real low-clearance reconstructed image obtaining a second total loss function; training a second variational autoencoder VAE2 based on a second total loss function, the training being for reducing a gap between a simulated low-resolution latent space and a real low-resolution latent space ​ inputting the real low-resolution image to be reconstructed into a trained end-to-end reconstruction model to obtain a reconstructed high-definition image comprises: inputting the real low-resolution image to be reconstructed into the second variational autoencoder VAE2 to obtain a low-resolution latent space of the real low-resolution image to be reconstructed; converting the low-resolution latent space of the real low-resolution image to be reconstructed to a high-definition latent space through the mapping network; processing the high-definition latent space through the first variational autoencoder VAE1 to obtain a reconstructed high-definition image.

2. The method of claim 1, wherein, the first variational autoencoder VAE1 comprises a first encoder and a first decoder, and the method comprises training the first variational autoencoder VAE1 based on at least the following steps: obtaining training sample data, wherein the training sample data comprises high-definition training data; based on the first encoder to encode real high definition images of high definition training data to the real high definition latent space ; based on the first decoder on the real high-definition latent space decoded to obtain a real high-definition reconstructed image ; based on the real high-definition image the real high-definition reconstructed image and the real high-definition latent space obtaining a first total loss function; training the first variational autoencoder VAE1 based on a first total loss function.

3. The method of claim 2, wherein, obtaining a first total loss function comprises: based on the real high-definition reconstructed image , real high-definition image obtaining a first reconstruction loss function; Based on real high-definition latent space , a first KL divergence loss function is obtained; based on the real high-definition reconstructed image , real high-definition image , the first variational autoencoder VAE1, obtain the first adversarial loss function; the sum of the first reconstruction loss function, the first KL divergence loss function, and the first adversarial loss function is the first total loss function.

4. The method of claim 1, wherein, obtaining a second total loss function comprises: obtaining a first sub-loss function, a second sub-loss function, and a fourth adversarial loss function, wherein the sum of the first sub-loss function, the second sub-loss function, and the fourth adversarial loss function is the second total loss function; wherein obtaining the first sub-loss function comprises: based on the simulated low-resolution reconstructed image , a simulated low-resolution image , obtaining a second reconstruction loss function; Based on analog low-clearance latent space , a second KL divergence loss function is obtained; based on the simulated low-resolution reconstructed image , a simulated low-resolution image , a second variational autoencoder VAE2, obtain a second adversarial loss function; the sum of the second reconstruction loss function, the second KL divergence loss function, and the second adversarial loss function is the first sub-loss function; obtaining the second sub-loss function comprises: based on the real low-resolution reconstructed image , real low-resolution image obtaining a third reconstruction loss function; based on the real low-resolution latent space a third KL divergence loss function is obtained; based on the real low-resolution reconstructed image , real low-resolution image , the second variational autoencoder VAE2, obtain a third adversarial loss function; the sum of the third reconstruction loss function, the third KL divergence loss function, and the third adversarial loss function is the second sub-loss function; based on the simulated low-resolution latent space , the real low-resolution latent space , a second variational autoencoder VAE2, obtains a fourth adversarial loss function.

5. The method of claim 1, wherein, training the mapping network comprises: based on the second encoder to encode the analog low definition image to an analog low definition latent space ; The simulated low-definition latent space converted to a high-definition latent space via a mapping network , by decoding through a second variational autoencoder VAE2, to obtain a reconstructed high-definition image ; based on the second variational autoencoder VAE2 encoding into the real high-definition latent space ; based on at least the real high-definition latent space , the reconstructed high-definition image , the simulated low-definition image , the first variational autoencoder VAE1, the second variational autoencoder VAE2 obtain a third total loss function; training the mapping network based on a third total loss function.

6. The method of claim 5, wherein, obtaining a third total loss function comprises: high-definition latent space converted through a mapping network , the real high-definition latent space , obtain a latent space distance loss function; based on the reconstructed high-definition image , the real high-definition image , a fifth adversarial loss is obtained; based on the reconstructed high-definition image , the real high-definition image , an activation function , obtaining a perception loss; the sum of the latent space distance loss function, the fifth adversarial loss, and the perceptual loss is the third total loss function.

7. An electronic device comprising a processor and a storage device, said storage device being adapted to store a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by the processor to execute the image super-resolution reconstruction method of any one of claims 1-6.

8. A computer readable storage medium having stored therein a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by the processor to execute the image super-resolution reconstruction method of any one of claims 1-6.

Images