Face image super-resolution method and system based on dual discriminant generative adversarial network

Abstract

The present invention discloses a facial image super-resolution method and device based on a dual discriminant generative adversarial network, which belongs to the field of facial image super-resolution. The method comprises: first, the generative adversarial network combines residual learning with dense connection learning to construct a dense-residual block, thereby enhancing the feature extraction capability without increasing the network depth and parameters; second, the discriminant network is improved, and a local discriminator is added to make the texture details of the reconstructed image more realistic and rich; finally, a new joint loss function is proposed. The network proposed by the present invention is superior to other latest facial image super-resolution algorithms, can generate higher quality facial images, and experiments on low-resolution facial datasets have proved the effectiveness of the algorithm.

Description

Technical Field

[0001] The present invention belongs to the technical field of super-resolution of facial images, and more specifically, relates to a method and device for super-resolution of facial images based on a dual-discriminant generative adversarial network. Background Art

[0002] Face super-resolution (SR) is a technique that infers a potential high-resolution (HR) image from an input low-resolution (LR) face image, which can significantly enhance the detail information of the LR face image. Therefore, it is widely used in face recognition, entertainment and other fields.

[0003] Although face SR is also classified as natural image SR, most natural images based on deep learning SR methods are not suitable for this case. Since there are many prior knowledge about face structure that is different from natural images, natural image SR methods cannot fully utilize the unique prior information of face images, which makes the face SR task different from the general natural image SR problem. In recent years, face priors have been widely used in existing face SR algorithms. For example, Song proposed a learning to hallucinate face images via component generation and enhancement (LCGE), which proved the role of face components in HR image reconstruction. In order to obtain a more realistic effect closer to the original high-resolution image, Yang et al. added an enhanced discriminative generative adversarial network for face super-resolution (EDGAN) to the generative network and applied it to the field of face image super-resolution. Yu et al. proposed a transformative autoencoder network to super-resolution very low-resolution unaligned and noisy face images (Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders, TDAE). Lu et al. proposed a parallel region-based deep residual network (PRDRN) for face hallucination, which uses region-based face hallucination to further learn accurate prior information.

[0004] At present, using neural network methods to learn the mapping relationship between low-resolution images and corresponding high-resolution images for image super-resolution reconstruction has become the mainstream method in this field. The core idea is to stack more convolutional layers to deepen the network structure and increase the receptive field of the convolution kernel to improve the feature extraction of high-frequency information of the image, thereby improving the quality of the reconstructed image. However, deepening the super-resolution reconstruction network structure will increase the number of model parameters, making the training of the model more difficult. Later, the model based on the generative adversarial network continuously generates images with clearer texture details in a generative adversarial manner, which has the advantage of improving the perceived quality of images in SR tasks. However, the reconstructed images often produce artifacts or even deformations in locations with rich texture details such as facial features. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention proposes a face image super-resolution method and device based on a dual discriminant generative adversarial network, thereby solving the technical problem that the texture details of the reconstructed image are not realistic and rich enough in the current face super-resolution reconstruction algorithm.

[0006] To achieve the above object, according to one aspect of the present invention, a face image super-resolution method based on a dual-discriminant generative adversarial network is provided, comprising:

[0007] (1) Generate adversarial networks by combining residual learning and dense connection learning to construct a dense-residual network;

[0008] (2) Add a local discriminator to the discriminant network to determine whether the image is real or fake. The local discriminator and the global discriminator share part of the network structure.

[0009] (3) Fusion the loss functions of the generative adversarial network and the discriminative network.

[0010] In some optional embodiments, step (1) comprises:

[0011] The RRDB network unit that combines a multi-layer residual network and a densely connected network is used to replace the BN network unit in the generative adversarial network to construct a dense-residual network.

[0012] In some optional embodiments, step (2) comprises:

[0013] The global discriminator distinguishes the real high-resolution face images and the face images generated by the dense-residual network G as a whole;

[0014] The local discriminator is used to locally distinguish the face image generated by G from the real high-resolution face image;

[0015] A global discriminator is used to estimate the probability that a real high-resolution face image is more realistic than a face image generated by the dense-residual network G, and a joint loss of the global discriminator and the local discriminator is used to constrain the dense-residual network G to synthesize a fake image that is more realistic than the real high-resolution face image.

[0016] In some optional implementation schemes, the global discriminator and the local discriminator both include N groups of network structures consisting of convolution layer-spectral normalization layer-activation function, and the values ​​of N in the global discriminator and the local discriminator are different. For the global discriminator, a relative discriminator with minimum square root loss is used.

[0017] In some optional embodiments, step (3) comprises:

[0018] The pixel-domain loss function in the generative adversarial network is used to constrain the difference between the super-resolution image and the original high-resolution image in the pixel domain space; the perceptual loss function in the discriminant network is used to constrain the difference in the feature domain space of the super-resolution image in the generative network; the adversarial loss function is used to constrain the adversarial learning of the generative network and the adversarial network.

[0019] According to another aspect of the present invention, a facial image super-resolution device based on a dual-discriminant generative adversarial network is provided, comprising:

[0020] A generation module, used to construct a dense-residual network by combining residual learning and dense connection learning through a generative adversarial network;

[0021] The image discrimination module is used to add a local discriminator to the discrimination network to judge whether the image is real or fake. The local discriminator and the global discriminator share part of the network structure.

[0022] The fusion module is used to fuse the loss functions of the generative adversarial network and the discriminative network.

[0023] In some optional implementation schemes, the generation module is used to replace the BN network unit in the generative adversarial network with the RRDB network unit that integrates a multi-layer residual network and a densely connected network to construct a dense-residual network.

[0024] In some optional embodiments, the image discrimination module is used to distinguish between real high-resolution face images and face images generated by a dense-residual network G as a whole by a global discriminator; to distinguish between face images generated by G and real high-resolution face images locally by a local discriminator; to use a global discriminator to estimate the probability that a real high-resolution face image is more real than a face image generated by a dense-residual network G, and to constrain the dense-residual network G to synthesize a fake image that is more real than a real high-resolution face image through a joint loss of the global discriminator and the local discriminator.

[0025] In some optional implementation schemes, the global discriminator and the local discriminator both include N groups of network structures consisting of convolution layer-spectral normalization layer-activation function, and the values ​​of N in the global discriminator and the local discriminator are different. For the global discriminator, a relative discriminator with minimum square root loss is used.

[0026] In some optional embodiments, the fusion module is used to generate a pixel domain loss function in the adversarial network to constrain the difference between the super-resolution image and the original high-resolution image in the pixel domain space; the perceptual loss function in the discriminant network is used to constrain the difference in the feature domain space of the super-resolution image in the generative network; and the adversarial loss function is used to constrain the adversarial learning of the generative network and the adversarial network.

[0027] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0028] A face super-resolution network based on Double Discriminant Generates Adversarial Network (DDGAN) is proposed. Firstly, the generative adversarial network combines residual learning and dense connection learning to construct dense-residual blocks, which enhances the feature extraction capability without increasing the network depth and parameters. Secondly, the discriminant network is improved by adding a local discriminator, making the texture details of the reconstructed image more realistic and rich. Finally, a new joint loss function is proposed. Experiments on low-resolution face datasets prove the effectiveness of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of a process of super-resolution of face images based on a dual-discriminant generative adversarial network provided by an embodiment of the present invention;

[0030] Figure 2 is a structural diagram of a discriminator module provided by an embodiment of the present invention;

[0031] Figure 3 is a structural diagram of an image discrimination module provided by an embodiment of the present invention;

[0032] Figure 4 is a result graph on a HHFQ data set provided by an embodiment of the present invention;

[0033] Figure 5 It is a structural diagram of RRDB in a generator G provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0035] Embodiment 1

[0036] The present invention proposes a face image super-resolution method based on a dual discriminant generative adversarial network. The face image super-resolution method uses a generative adversarial network to combine residual learning with dense connection learning to construct a dense-residual block, thereby enhancing the feature extraction capability without increasing the network depth and parameters to obtain a better image.

[0037] Figure 1 : This is a flow chart of a face image super-resolution method based on a dual-discriminant generative adversarial network provided by an embodiment of the present invention. The method is mainly implemented through the following three steps:

[0038] S1: Generate adversarial networks by combining residual learning and dense connection learning to construct dense-residual networks;

[0039] In the embodiment of the present invention, step S1 can be implemented in the following manner:

[0040] like Figure 5 As shown in the figure, by using the RRDB (Rsidual in Residual Dense Block) structure, the dense connection network and the residual network are perfectly combined. Without increasing the network depth and computational overhead, the deep and shallow features of the LR image are combined to the maximum extent, while avoiding the gradient vanishing problem caused by the network being too deep. At the same time, compared with the common super-resolution network, the BN (Batch normalization) layer is not used here. In the residual network, each layer is connected to the previous layer (usually 2 to 3 layers) through a jump connection, and the connection method is element-level addition. During the training of the generative adversarial network, the BN layer in the generative adversarial network uses the mean and variance of each batch of data in the batch to normalize the features, but after any image passes through the BN layer, its color distribution will be normalized. In other words, it destroys the original contrast information of the image, so the addition of the BN layer will affect the SR result and cause unpredictable artifacts. In order to ensure the reconstruction performance, the BN layer is removed from the original network.

[0041] S2: Improve the discriminant network and add a local discriminator to enhance the feature extraction capability without increasing the network depth and parameters, making the texture details of the reconstructed image more realistic and rich;

[0042] Among them, the local discriminator and the global discriminator share some network structures, and their essential differences are: the way of calculating the similarity between the real picture and the generated picture is different. Among them, the global discriminator compares the whole picture; the local discriminator divides the picture into N*N blocks, calculates the similarity of each block separately, and generates a similarity matrix. This global-local structure ensures that all local patches of the reconstructed image look like real normal patches, which has been proven to be very useful for super-resolution reconstruction of faces with rich structural and texture details.

[0043] In the embodiment of the present invention, step S2 is mainly to distinguish whether an image is real or fake, including the following steps: first, a global discriminator is used to distinguish the real high-resolution face image and the face image generated by the dense-residual network G as a whole; similarly, a local discriminator is used to locally distinguish the image generated by G and the real high-resolution face image, and finally a global discriminator is used to estimate the probability that the real high-resolution face image is more real than the face image generated by the dense-residual network G, and a joint loss of the global discriminator and the local discriminator is used to constrain the dense-residual network G to synthesize a fake image that is more real than the real image.

[0044] Specifically, Figure 2 As shown, N represents the number of groups corresponding to the "convolution layer-spectral normalization layer-activation function" structure. Distinguish the real high-resolution face images and the face images generated by G from the whole; similarly, distinguish the images generated by G from the real high-resolution face images locally. For the global discriminator, a relativistic discriminator with the least-square loss is used, which estimates the probability that the real high-resolution face image is more realistic than the face image generated by the dense-residual network G, and guides G to synthesize a fake image that is more realistic than the real image. A local discriminator network is introduced to improve the quality of texture details in the generated images. By extracting randomly cropped local patches from the reconstructed and real high-resolution face images, and learning to distinguish whether they are real (HR images) or fake (G-generated images).

[0045] In the convolution layer, k, s, p, and n represent the size of the convolution kernel, stride, padding, and the number of convolution kernels, respectively. The size of all convolution kernels for the global discriminator and the local discriminator is selected to be 4. The number of convolution kernels in the convolution layer increases as the network deepens, starting from 64 and increasing by multiples of 2 to 512. When N=5 in the figure, the discriminator is a global discriminator, and the output result is a value; when N=3 in the figure, the discriminator is a local discriminator, and the result is a matrix. A value in the matrix represents a true and false matrix of a local block of size 70×70. The network structure is as follows Figure 3 shown.

[0046] S3: Finally, a new joint loss function is proposed, and experiments on a low-resolution face dataset demonstrate the effectiveness of the algorithm.

[0047] In the embodiment of the present invention, step S3 can be implemented in the following manner:

[0048] The loss based on the dual discriminant generative adversarial network consists of three parts: the pixel-domain loss function in the generative network is used to constrain the difference between the super-resolution image and the original high-resolution image in the pixel domain space; the perceptual loss function in the discriminant network is used to constrain the difference in the feature domain space of the super-resolution image in the generative network; the adversarial loss function is used to constrain the adversarial learning of the generative network and the adversarial network.

[0049] The present invention is compared with other image super-resolution reconstruction algorithms, and experimental data are provided to demonstrate the effectiveness of the present method. The parameters of the comparative experimental results are shown in Table 1 below. In order to ensure a fair comparison, we use the same training and test data sets to retrain and test the above networks. The best results are marked with bold fonts, and the suboptimal results are marked with underlines. Figure 4 shown.

[0050] Table 1

[0051]

[0052]

[0053] It can be seen from the experimental data in the above table that the methods of the present invention have obtained higher scores than the comparison methods, that is, they are better than the comparison algorithms.

[0054] Embodiment 2

[0055] In another embodiment of the present application, a face image super-resolution device based on a dual-discriminant generative adversarial network is provided, comprising:

[0056] A generation module, used to construct a dense-residual network by combining residual learning and dense connection learning through a generative adversarial network;

[0057] The image discrimination module is used to add a local discriminator to the discrimination network to judge whether the image is real or fake. The local discriminator and the global discriminator share part of the network structure.

[0058] The fusion module is used to fuse the loss functions of the generative adversarial network and the discriminative network.

[0059] The image discrimination module includes a discriminator module composed of a generative adversarial network module and two discriminator networks connected in parallel, and a fusion module is provided after the generative adversarial network module and the discriminator module. The fusion module includes three layers of sub-networks, and different sub-networks use different convolution kernels.

[0060] In some optional implementation schemes, the generation module is used to replace the BN network unit in the generative adversarial network with the RRDB network unit that integrates a multi-layer residual network and a densely connected network to construct a dense-residual network.

[0061] In some optional embodiments, an image discrimination module is used to globally distinguish between real high-resolution face images and face images generated by a dense-residual network G by a global discriminator; locally distinguish between face images generated by G and real high-resolution face images by a local discriminator; use a global discriminator to estimate the probability that a real high-resolution face image is more real than a face image generated by a dense-residual network G, and use a joint loss of the global discriminator and the local discriminator to constrain the dense-residual network G to synthesize a fake image that is more real than the real high-resolution face image.

[0062] In some optional implementation schemes, the global discriminator and the local discriminator both include N groups of network structures consisting of convolution layer-spectral normalization layer-activation function, and the values ​​of N in the global discriminator and the local discriminator are different. For the global discriminator, a relative discriminator with minimum square root loss is used.

[0063] In some optional implementation schemes, a fusion module is used to generate a pixel-domain loss function in the adversarial network to constrain the difference between the super-resolution image and the original high-resolution image in the pixel-domain space; the perceptual loss function in the discriminant network is used to constrain the difference in the feature domain space of the super-resolution image in the generative network; and the adversarial loss function is used to constrain the adversarial learning of the generative network and the adversarial network.

[0064] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0065] Those skilled in the art will readily appreciate that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A face image super-resolution method based on dual discriminant generative adversarial network, characterized in that: include: (1) Generate adversarial networks by combining residual learning and dense connection learning to construct a dense-residual network; (2) Add a local discriminator to the discriminant network to determine whether the image is real or fake. The local discriminator and the global discriminator share part of the network structure. (3) Fusion of the loss functions of the generative adversarial network and the discriminative network; Step (1) comprises: Use the RRDB network unit that combines a multi-layer residual network and a densely connected network to replace the BN network unit in the generative adversarial network to construct a dense-residual network; Step (2) comprises: The global discriminator distinguishes the real high-resolution face images and the face images generated by the dense-residual network G as a whole; The local discriminator is used to locally distinguish the face image generated by G from the real high-resolution face image; Use a global discriminator to estimate the probability that a real high-resolution face image is more realistic than a face image generated by the dense-residual network G, and constrain the dense-residual network G through the joint loss of the global discriminator and the local discriminator to synthesize a fake image that is more realistic than the real high-resolution face image; Both the global discriminator and the local discriminator include N groups of network structures consisting of convolutional layer-spectral normalization layer-activation function, and the value of N in the global discriminator and the local discriminator is different. For the global discriminator, a relative discriminator with minimum square root loss is used; Step (3) includes: The pixel-domain loss function in the generative adversarial network is used to constrain the difference between the super-resolution image and the original high-resolution image in the pixel domain space; the perceptual loss function in the discriminant network is used to constrain the difference in the feature domain space of the super-resolution image in the generative network; the adversarial loss function is used to constrain the adversarial learning of the generative network and the adversarial network.

2. A face image super-resolution device based on a dual discriminant generative adversarial network, characterized in that: include: A generation module is used to construct a dense-residual network by combining residual learning and dense connection learning from a generative adversarial network; The image discrimination module is used to add a local discriminator to the discrimination network to judge whether the image is real or fake. The local discriminator and the global discriminator share part of the network structure. The fusion module is used to fuse the loss functions of the generative adversarial network and the discriminative network; The generation module is used to replace the BN network unit in the generative adversarial network with the RRDB network unit that integrates the multi-layer residual network and the dense connection network to construct a dense-residual network; The image discrimination module is used to use a global discriminator to distinguish the real high-resolution face image and the face image generated by the dense-residual network G from the whole; use a local discriminator to locally distinguish the face image generated by G from the real high-resolution face image; use the global discriminator to estimate the probability that the real high-resolution face image is more real than the face image generated by the dense-residual network G, and use the joint loss of the global discriminator and the local discriminator to constrain the dense-residual network G to synthesize a fake image that is more real than the real high-resolution face image; Both the global discriminator and the local discriminator include N groups of network structures consisting of convolutional layer-spectral normalization layer-activation function, and the value of N in the global discriminator and the local discriminator is different. For the global discriminator, a relative discriminator with minimum square root loss is used; The fusion module is used to generate a pixel domain loss function in the adversarial network to constrain the difference between the super-resolution image and the original high-resolution image in the pixel domain space; the perceptual loss function in the discriminant network is used to constrain the difference in the feature domain space of the super-resolution image in the generated network; and the adversarial loss function is used to constrain the adversarial learning of the generated network and the adversarial network.

Images