Image restoration method and device, electronic equipment, readable storage medium and chip

Abstract

The invention provides an image restoration method and device, electronic equipment, a readable storage medium and a chip, and the method comprises the steps: obtaining a restoration algorithm of a generative model and a first fuzzy face image, the restoration algorithm of the generative model comprising a discriminator, an encoder and a decoder; extracting image features of the first fuzzy face image through an encoder; calculating hidden variables of the image features through a restoration algorithm of the generative model; inputting the hidden variables into a decoder, and performing up-sampling through the decoder to obtain a recovery result; and classifying the restoration results according to a discriminator, and determining a clear face image. According to the scheme of the invention, the variational auto-encoder and the generative adversarial network are combined to perform deblurring restoration on the face image, and through a mode of adding the discriminator of the generative adversarial network in the structure of the variational auto-encoder and combining the advantages of the variational auto-encoder and the discriminator of the generative adversarial network, the convergence speed of face image restoration is higher, and the restoration result is clearer.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image restoration method, device, electronic device, readable storage medium and chip. Background Art

[0002] Due to insufficient light during the shooting process, the original camera's weak shooting ability, and the camera's relative shaking during exposure, the images captured by the camera often have various blur problems. Blurring is caused by the change in the relative position of the camera and the scene system or target during the sensor exposure process. During the exposure stage of the shooting process, if the camera is not held steady and shakes, or the subject moves quickly, the shaking itself will cause motion blur in the image. The greater the degree of shaking, the higher the degree of blur, and the faster the shaking speed, the blurrier the image.

[0003] The image blur degradation model is as follows: Figure 1 As shown, I S represents a clear and sharp face image without degradation, I B It represents a degraded blurred face image, which is a low-quality distorted image after being disturbed by additive noise and convolved with the degradation operator K. The degradation operator K, also known as the blur kernel, describes the mapping relationship and distribution of the real information on the clear image on the blurred image. In most cases, the camera shake and the movement of the photographed target are very fast and random and unpredictable, so the blur kernel information cannot be obtained in advance. Additive noise η also has a great influence on image degradation. Sometimes in the deconvolution deblurring task, due to the presence of random noise, the image is more likely to have problems such as boundary effect and checkerboard effect. The mathematical model of blurred face image can be expressed as follows:

[0004]

[0005] Without knowing the blur kernel K, we can get the degraded blurred face image I from the given B To get a clear face image I S It is relatively difficult, and the error caused in the process of blur kernel estimation will be amplified in the subsequent deconvolution process. Therefore, it is difficult to restore a clear face image in a blurry face image with noise or occlusion, and the clarity of the restored face image is low. Summary of the invention

[0006] In view of this, the present invention aims to solve the problem of low clarity of a facial image restored from a blurred facial image with noise or occlusion.

[0007] Specifically, the present invention is achieved through the following technical solutions:

[0008] A first aspect of the present invention provides an image restoration method.

[0009] A second aspect of the present invention provides an image restoration device.

[0010] A third aspect of the present invention provides an electronic device.

[0011] A fourth aspect of the present invention provides a readable storage medium.

[0012] A fifth aspect of the present invention provides a chip.

[0013] The image restoration method provided by the present invention comprises: obtaining a restoration algorithm of a generative model and a first blurred face image, wherein the restoration algorithm of the generative model comprises a discriminator, an encoder and a decoder; extracting image features of the first blurred face image through the encoder; calculating latent variables of the image features through the restoration algorithm of the generative model; inputting the latent variables into the decoder, performing upsampling through the decoder to obtain a restoration result; and classifying the restoration result according to the discriminator to determine a clear face image.

[0014] In some technical solutions, optionally, before obtaining the restoration algorithm of the generative model and the first blurred face image, it also includes: obtaining a second blurred face image, the second blurred face image includes a blur kernel; based on the process parameters of the restoration algorithm of the generative model, setting a generative adversarial network and a variational autoencoder, and transmitting the second blurred face image to the variational autoencoder, the variational autoencoder includes an encoder and a decoder, and the generative adversarial network includes a discriminator; wherein the discriminator uses a shallow convolutional neural network as a classifier; and determining the restoration algorithm of the generative model according to the way the generative adversarial network and the variational autoencoder process the second blurred face image.

[0015] In some technical schemes, optionally, a restoration algorithm of a generative model is determined based on a generative adversarial network and a variational autoencoder, including: obtaining a convolution residual block; down-convolving the second blurred face image through an encoder to reduce the image size, and inputting it into the convolution residual block to determine the first image; using a neural network to calculate the mean μ(I) and standard deviation σ(I) of the first image; sampling ε from a standard normal distribution N(0, 1), and determining a latent variable z through the formula z=μ(I)+σ(I)·ε; up-sampling the latent variable z through a decoder and then convolving it, and inputting it into the convolution residual block to determine the second image; obtaining a true value sample corresponding to the second blurred face image, and inputting the true value sample and the second image into a discriminator to determine the training set result; and determining the restoration algorithm of the generative model based on the training set result.

[0016] In some technical solutions, optionally, obtaining a second blurred face image includes: obtaining a clear face image as a sample from a public face image database; simulating a blur kernel by means of Gaussian random noise; and determining the second blurred face image based on convolution synthesis of the blur kernel and the sample.

[0017] In some technical solutions, optionally, the convolution residual block includes: a convolution layer, an instance normalization layer and a linear activation function, and the convolution layer is set with a convolution kernel size parameter, a step parameter and a zero padding parameter.

[0018] In some technical solutions, optionally, the encoder includes: a convolution layer, an instance normalization layer and a linear activation function, the convolution layer is set with a convolution kernel size parameter, a step parameter and a zero padding parameter; the structure of the decoder is the same as that of the encoder.

[0019] The second aspect of the present invention provides a device, including: an acquisition module, used to acquire the restoration algorithm of a generative model and a first blurred face image, the restoration algorithm including a discriminator, an encoder and a decoder; an extraction module, used to extract the image features of the first blurred face image through the encoder; a calculation module, used to calculate the latent variables of the image features through the restoration algorithm of the generative model; a restoration module, used to input the latent variables into the decoder, and obtain the restoration result by upsampling through the decoder; an image generation module, used to classify the restoration result according to the discriminator to determine a clear face image.

[0020] An embodiment of the third aspect of the present invention provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction implements the steps in the first aspect when executed by the processor.

[0021] An embodiment of the fourth aspect of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the first aspect are implemented.

[0022] An embodiment of the fifth aspect of the present invention provides a chip, the chip includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps in the first aspect.

[0023] The technical solution provided by the present invention brings at least the following beneficial effects:

[0024] The present invention proposes an image restoration method, which combines a variational autoencoder with a generative adversarial network to deblur and restore facial images. By adding a discriminator of the generative adversarial network to the structure of the variational autoencoder and combining their respective advantages, the convergence speed of facial image restoration is faster and the restoration result is clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0027] Figure 1 A schematic block diagram of an image blur degradation model in a related technical solution provided by an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of a flow chart of an image restoration method provided by an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of a partial flow chart of an image restoration method provided by an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of a partial flow chart of an image restoration method provided by an embodiment of the present invention;

[0031] Figure 5 A schematic diagram of a partial flow chart of an image restoration method provided by an embodiment of the present invention;

[0032] Figure 6 A schematic block diagram of the structure of an image restoration device provided by an embodiment of the present invention;

[0033] Figure 7 A schematic block diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0034] Figure 8 A schematic block diagram of the network structure of a variational autoencoder provided in an embodiment of the present invention;

[0035] Fig. 9 A schematic block diagram of a network structure of a residual block provided in an embodiment of the present invention;

[0036] Fig.10 A schematic block diagram of the network structure of a discriminator provided in an embodiment of the present invention;

[0037] Fig.11 A schematic block diagram of a facial image deblurring process provided by an embodiment of the present invention.

[0038] in, Figure 6 and Figure 7 The corresponding relationship between the component names and numbers in is as follows:

[0039] 900: image restoration device; 902: acquisition module; 904: extraction module; 906: calculation module; 908: restoration module; 910: image generation module; 1000: electronic device; 1109: memory; 1110: processor. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figure 2 The first aspect of the present invention provides an image restoration method, comprising the following steps:

[0042] Step S102: obtaining a restoration algorithm of a generative model and a first blurred face image, wherein the restoration algorithm of the generative model includes a discriminator, an encoder, and a decoder;

[0043] Step S104: extracting image features of the first blurred face image through an encoder;

[0044] Step S106: Calculate latent variables of image features through a restoration algorithm of a generative model;

[0045] Step S108: input the latent variable into the decoder, and perform upsampling through the decoder to obtain a restored result;

[0046] Step S110: classify the restoration results according to the discriminator to determine a clear face image.

[0047] According to the image restoration method provided by the present invention, it is proposed to use a generative model to achieve the restoration task of the face image in order to solve the problem of motion blur in the face image. That is, a variational autoencoder is combined with a generative adversarial network to deblur the face image. The discriminator of the generative adversarial network is connected to the structure of the variational autoencoder, and the restoration algorithm of the generative model is combined to improve the clarity of the restored blurred face image. Specifically, a restoration algorithm of a generative model and a blurred face image to be restored (a first blurred face image) are obtained, wherein the restoration algorithm of the generative model is generated by combining a variational autoencoder and a generative adversarial network; the core of a variational autoencoder (VAE) includes an encoder network and a decoder network, wherein the encoder part is used to learn the mapping from input data to an implicit space, in which the encoder compresses the input data into a point in a latent space, the coordinates of which are represented by a latent vector, which is not a definite point but a distribution, usually assumed to be a normal distribution, which is determined by a mean and a standard deviation; the decoder part is used to learn the reverse mapping from the implicit space to the input data, the decoder samples from the latent vector generated by the encoder, and then attempts to decode the sampled point into the original input data. In the image restoration process, the variational autoencoder enables the restoration result to have better low-frequency information, i.e., information such as the face background contour. Generative Adversarial Network (GAN) is a deep learning model, which includes two parts: a generator and a discriminator. By training the generator network and the discriminator network to confront each other, the generator network generates and updates false data values ​​and inputs them to the discriminator network, so that the discriminator continuously improves the ability to distinguish true data from false data. Usually, the generator generates false data or noisy images by machine, with the purpose of making it difficult for the discriminator to distinguish the difference between true data and false data, and the discriminator is trained by training sets, continuously improving the ability to recognize true data, and outputting true data determined by the discriminator. In the present invention, only the discriminator in the generative adversarial network is used, the result of the variational autoencoder output is used as the input, the variational autoencoder output result and the real clear image are trained, and finally a clear face image is output. The discriminator of the generative adversarial network has a better restoration effect for high-frequency information such as texture details. The discriminator, encoder and decoder are combined to generate a restoration algorithm and a restoration algorithm of a generative model. The image features of the first blurred face image are extracted by the encoder in the variational autoencoder, and the latent variables corresponding to the image features are calculated by the restoration algorithm of the generative model. The latent variables are then upsampled in the decoder corresponding to the input value of the latent variables to achieve image reconstruction, and the reconstructed result is input to the discriminator for judgment until the discriminator determines that the reconstructed result is true data and outputs it to determine it as a restored clear face image.

[0048] It can be understood that by introducing the discriminator in the generative adversarial network into the variational autoencoder to combine with the restoration algorithm of the determined generative model, the variational autoencoder is combined with the advantages of better low-frequency information in the image restoration process and the discriminator of the generative adversarial network has better restoration effects on high-frequency information such as texture details, thereby improving the clarity of the restoration algorithm of the generative model for blurred face images.

[0049] In some embodiments, optionally, Figure 3 As shown, before obtaining the restoration algorithm of the generative model and the first blurred face image, it also includes:

[0050] Step S1002: obtaining a second blurred face image, where the second blurred face image includes a blur kernel;

[0051] Step S1004: Based on the process parameters of the restoration algorithm of the generative model, a generative adversarial network and a variational autoencoder are set, and the second blurred face image is transmitted to the variational autoencoder, where the variational autoencoder includes an encoder and a decoder, and the generative adversarial network includes a discriminator;

[0052] Step S1006: Determine a restoration algorithm of a generative model according to the way in which the generative adversarial network and the variational autoencoder process the second blurred face image.

[0053] In this embodiment, before restoring the clarity of the blurred face image to be restored (the first blurred face image), a second blurred face image for training should be obtained, wherein the second blurred face image is a blurred face image including a blur kernel and the blur kernel is added and determined by the operator, so the blur kernel of the second blurred face image is a known condition. The restoration algorithm of the generative model composed of the variational autoencoder and the generative adversarial network is trained by at least one second blurred face image, that is, the second blurred face image is transmitted to the variational autoencoder, the second blurred face image is restored to clarity by the encoder and decoder in the variational autoencoder, and then the discriminator in the generative adversarial network distinguishes the face image after the clarity restoration and the corresponding clear face image, and outputs the clear face image determined by the restoration algorithm of the generative model. The generative adversarial network and the variational autoencoder perform deep learning on the clarity restoration of at least one second blurred face image, thereby determining the process parameters of the restoration algorithm of the generative model, and obtaining the restoration algorithm of the generative model for restoring the first blurred face image, wherein the process parameters are the learning parameters corresponding to the discriminator in the generative adversarial network and the encoder and decoder of the variational autoencoder, respectively. Among them, the discriminator adopts a shallow convolutional neural network (CNN) as a classifier.

[0054] It can be understood that by using the second blurred face image with a known blur kernel as a training set, the variational autoencoder and the generative adversarial network are trained to obtain the restoration algorithm of the generative adversarial model, thereby improving the discrimination efficiency of the discriminator corresponding to the generative adversarial network and reducing the time it takes for the discriminator to distinguish the authenticity of the image data transmitted from the variational autoencoder.

[0055] In some embodiments, optionally, Figure 4 As shown, the restoration algorithm of the generative model is determined based on the generative adversarial network and the variational autoencoder, including:

[0056] Step S10062: Obtain a convolution residual block;

[0057] Step S10064: down-convolve the second blurred face image through the encoder to reduce the image size, and input it into the convolution residual block to determine the first image;

[0058] Step S10066: Calculate the mean μ(I) and standard deviation σ(I) of the first image using a neural network;

[0059] Step S10068: Sample ε from the standard normal distribution N(0,1), and determine the latent variable z by the formula z=μ(I)+σ(I)·ε;

[0060] Step S10070: upsampling the latent variable z through the decoder and then convolving it, and inputting it into the convolution residual block to determine the second image;

[0061] Step S10072: obtaining a true value sample corresponding to the second blurred face image, inputting the true value sample and the second image into the discriminator, and determining the training set result;

[0062] Step S10074: Determine the restoration algorithm of the generative model according to the training set results.

[0063] In this embodiment, the algorithm flow corresponding to the restoration algorithm of the generative model generated by the generative adversarial network and the variational autoencoder is as follows: obtain a convolution residual block (ResBlock), which helps to achieve faster convergence in the image convolution process and improve the reconstructed face image. The second blurred face image (i.e., the blurred face image used for training, the blur kernel is known) is input to the variational autoencoder, and the encoder in the variational autoencoder convolves the second blurred face image downward to reduce the image size, and inputs the reduced image into the convolution residual block to determine it as the first image. Among them, no pooling is performed after convolution, and pooling will cause the loss of image features, and reducing the image size is achieved by setting the step size of the convolution layer to 2. The neural network is used to calculate the mean μ(I) and standard deviation σ(I) of at least one second blurred face image after the image is reduced in size (i.e., the first image); the process of calculating the hidden variable z draws on the variational autoencoder re-parameterization method, and samples ε from the standard normal distribution N(0, 1). By drawing on the variational autoencoder re-parameterization method, random sampling does not need to participate in gradient descent, and only needs to update the sampling result. The hidden variable z is determined by the formula z=μ(I)+σ(I)·ε; the decoder convolves the determined hidden variable z after upsampling, reconstructs a clear face image from the hidden variable and inputs it into the convolution residual block to determine the second image (i.e., the clear face image output by the variational autoencoder), and inputs the second image and the clear face image corresponding to the second image as true value samples into the discriminator of the generative adversarial network, determines the training set result obtained by training the second blurred face image, and determines the restoration algorithm of the generative model based on at least one training.

[0064] In a specific embodiment, the schematic diagram of the process of deblurring a face image using a restoration algorithm of a generative model is as follows: Fig.11 As shown, input fuzzy face image I B To the variational autoencoder (VAE1), through the encoder G in the variational autoencoder VAE1 1 , encoder G 1 For blurred face image I B After down-convolution reduces the image size, the reduced image is input into the residual convolution block, and the neural network calculates μ(I)σ(I), i.e., the mean and standard deviation, samples ε from N(0, 1), and determines the latent variable z by the formula z=μ(I)+σ(I)·ε, which is then input into the decoder E 1 Medium Output I G (i.e. clear face image).

[0065] In a specific embodiment, the network structure of the discriminator in the generative adversarial network is as follows: Fig.10As shown in the figure, after the variational autoencoder reduces the dimension of the blurred face image through the convolutional network and restores it to a clear face image, the image is straightened and input into a shallow convolutional neural network (CNN). The shallow convolutional neural network acts as a classifier corresponding to the discriminator to classify the clear face image output by the variational autoencoder to determine whether it is true or false, where the true corresponding clear face image is a clear face image that meets the clarity requirement, and the false corresponding restoration result does not meet the clarity requirement. Shallow convolutional neural networks extract local features of input data through convolution operations, and form complex feature representations through multi-layer convolution and pooling operations, and finally perform tasks such as classification or regression through fully connected layers. By using shallow convolutional neural networks as classifiers corresponding to discriminators, neurons in the convolution layer are only connected to a local area of ​​the input data (i.e., the local receptive field), which helps to capture the local features of the restored face image; the same convolution kernel shares weights at all positions of the input data, which greatly reduces the number of network parameters and reduces the complexity of the model; and no matter where the features in the input data appear, the convolution operation can extract the same features, which makes the discriminator more efficient and accurate when processing image data.

[0066] Specifically, the discriminator uses CNN as the classifier, the input is the second blurred image and the true value of the clear image generated by the variational autoencoder, and the output is "true" or "false". Among them, the loss function selects the L1 loss function in order to better restore the low-frequency information of the image. The loss function formula is as follows:

[0067]

[0068] Among them, C is the number of channels of the image, W is the width of the image, H is the height of the image, and I S is the generated second blurred image, I G is the true value image (i.e., clear face image), i, j, k are all pixel values ​​of the image.

[0069] The images restored by the variational autoencoder include: the input image is 3×128×128, using a 3×3 convolution kernel, generating a 32×64×64 image after the first convolution, using the LeakyReLU activation function after convolution, generating a 32×16×16 value after the second convolution, using the LeakyReLU activation function after convolution, generating a 64×8×8 image after the third convolution, using the LeakyReLU activation function after convolution, and then inputting it into the neural network of the fully connected layer, with 4096 neurons in the first layer, 1024 neurons in the second layer, and 1 neuron in the third layer. After the output, the sigmoid activation function is used to output a scalar between 0 and 1 to represent the probability of the second blurred image being similar to the true value. If it is close to 1, it outputs "true", and if it is close to 0, it outputs "false".

[0070] In some embodiments, optionally, Figure 5 As shown, obtaining a second blurred face image includes:

[0071] Step S10022: Obtain a clear face image as a sample from a public face image database;

[0072] Step S10024: simulating blur kernel by means of Gaussian random noise;

[0073] Step S10026: Determine a second blurred face image based on the blur kernel and sample convolution synthesis.

[0074] In this embodiment, a clear face image is obtained through a public face image data set, and a blur kernel is simulated by Gaussian random noise. Gaussian random noise is also called white noise, which is a random noise that obeys a normal distribution. In deep learning, Gaussian noise is often added to the input data during training to improve the robustness and generalization ability of the model. By adding noise to the input data, the model is forced to learn features that are robust to small changes in the input, which makes the second blurred face image perform better as a training set. Since the camera shake and the movement of the photographed target are very rapid and random and unpredictable, it is impossible to obtain blur kernel information in advance. Without knowing the blur kernel, it is difficult to obtain a clear face image from a given degraded blurred face image, and the error caused in the process of blur kernel estimation will be amplified in the subsequent deconvolution process, so it is difficult to restore a clear face image in a blurred face image with noise or occlusion. By convolving a clear image with a blur kernel in the present invention, an artificially synthesized blur kernel for a blurred image is obtained, and a blurred face image is synthesized by convolution of the blur kernel and a face image, so that the blur kernel information between the clear face image and the second blurred face image is determined, thereby improving the accuracy of using the second blurred face image as a training set for the encoder and decoder in the variational autoencoder and the discriminator in the generative adversarial network, thereby improving the clarity of the restoration algorithm of the generative model for the blurred face image.

[0075] In some embodiments, optionally, the convolution residual block includes: a convolution layer, an instance normalization layer and a linear activation function, and the convolution layer is set with a convolution kernel size parameter, a step parameter and a zero padding parameter.

[0076] In this embodiment, the network structure of the convolutional residual block is as follows: Fig. 9 As shown, it includes a convolution layer, an instance normalization layer and a linear activation function. Specifically, the network structure is a convolution layer, a first instance normalization layer, a linear activation function, a convolution layer and a second instance normalization layer from top to bottom. Among them, the convolution layer uses a convolution kernel of size 3×3 (i.e., the convolution kernel size parameter is 3×3), a step size of 1 (i.e., the step size parameter is 1), a zero padding item of 0 (i.e., the zero padding item parameter is 0), and does not change the channels before and after the convolution layer. The instance normalization layer is mainly used to calculate the variance and mean of the data in each channel. After calculation, the output data is set to a parameter with a mean of 0 and a variance of 1, so as to more effectively learn the features of the data. Since each convolution will change the channel and size of the image, normalization is performed after the convolution to improve the original effect of the image. The purpose of the residual block is to perform residual learning and help the network converge. Therefore, the image size and the number of channels are not changed after the residual block.

[0077] In some embodiments, optionally, the encoder includes: a convolution layer, an instance normalization layer and a linear activation function, the convolution layer is set with a convolution kernel size parameter, a step parameter and a zero padding parameter; the structure of the decoder is the same as that of the encoder.

[0078] In this embodiment, the network structure of the variational autoencoder is as follows Figure 8 As shown, reducing the image size by down-convolution through a variational autoencoder includes: obtaining a blurred image (i.e., a second blurred face image) with a pixel size of 128×128, inputting it into the encoder, wherein the channels in the encoder include: a convolution kernel (3×3Conv), an instance normalization layer (InstanceNorm), and a linear activation function (ReLU), and down-convolving the second blurred face image through a pixel size of 64×64 channel, a pixel size of 32×32 channel, and a pixel size of 16×16 channel, wherein a convolution residual block (ResBlock) is provided between each channel, and the image after each convolution is converged and improved through the convolution residual block, and after down-convolution through the three channels of the encoder, the mean μ(I) and the standard deviation σ(I) of the reduced-size image are calculated through a neural network, and according to the formula z=μ(I)+σ (I)·ε determines the latent variable z; reconstructing the image by up-sampling convolution through the variational autoencoder includes: inputting the latent variable z into the decoder, the channels in the decoder include: dual-channel convolution kernel (Bilinear+Conv), instance normalization layer (InstanceNorm) and linear activation function (ReLU), and up-convolving the latent variable z after sampling through pixel size 16×16 channels, pixel size 32×32 channels and pixel size 64×64 channels, and setting a convolution residual block (ResBlock) between each channel, converging and improving the image after each convolution through the convolution residual block, and after up-convolution through the three channels of the decoder, the hyperbolic tangent function (tanh) is used as the activation function to obtain the deblurred result (i.e., the restored face image) of the reconstructed pixel size 128×128. Among them, the structure of the decoder is the same as that of the encoder, except that the convolution kernel parameters are changed, and the tanh function is used as the activation function in the decoder.

[0079] In a specific embodiment, the algorithm flow includes: Step 1: First, blur the image I B Input to the variational autoencoder, through the encoder G 1 Convolve the image downward to reduce the image size. Use a neural network to calculate the mean μ(I) and standard deviation σ(I) of each input image after the image is reduced in size; Step 2: The process of calculating the hidden variable draws on the variational autoencoder re-parameterization method, sampling ε from the standard normal distribution N(0, 1), and calculating the dependent variable z through the formula z=μ(I)+σ(I)·ε; Step 3: Decoder E 1The input latent variable z is upsampled and then convolved to reconstruct a clear face image I from the latent variable G , and then generate the face image I G The ultimate goal of face reconstruction is S Input to the discriminator D 1 The discriminator structure adopts a shallow convolutional neural network.

[0080] The convolutional neural network structure of the variational autoencoder of the algorithm proposed in this embodiment is as follows: Figure 8 As shown in the figure, the structure of the encoder is similar to a style transfer task. It includes three convolution layers with a convolution kernel size of 3×3, a stride of 2, and a zero padding of 1. Each convolution layer is followed by an instance normalization layer (InstanceNorm Layer) and a linear activation function Relu. Pooling is not performed after convolution, because pooling may cause the loss of image features. The convolutional neural network used in the present invention reduces the image size by setting the stride of 2. The expression of the instance normalization layer is as follows:

[0081]

[0082] Among them, y i，j is the image pixel value after normalization of a single channel, x i，j is the pixel value of a single channel input image, ζ is random noise, μ and σ 2 are the mean and variance respectively, and the calculation formulas are as follows:

[0083]

[0084]

[0085] Among them, H is the length of the image, W is the width of the image, i is the pixel value in the length direction, and j is the pixel value in the width direction.

[0086] The instance normalization layer acts on a single image and is suitable for generating models. In the face image deblurring task, each pixel contains the information of the blur kernel, so instance normalization is selected instead of batch normalization. Therefore, batch normalization considers all images in a batch, which will cause the unique details in each image to be lost. Instance normalization normalizes the length and width of each image, while ensuring the independence of each image instance.

[0087] After the function input image passes through the convolution part, it enters the convolution residual block. The convolution residual block helps to achieve faster convergence and improve the reconstructed face image. The structure of the convolution residual block is as follows Fig. 9As shown in the figure, the convolution layer of the convolution residual block uses a 3×3 convolution kernel, a step size of 1, a zero-filling term of 0, and does not change the channels before and after the convolution. The purpose of the convolution residual block is to perform residual learning and help the network converge. Therefore, the image size and the number of channels are not changed after the image passes through the residual block.

[0088] The structure of the decoder is similar to that of the encoder. The decoder upsamples the input latent variables to achieve image reconstruction. The upsampling method uses the nearest neighbor interpolation method plus convolution. The nearest neighbor difference doubles the size of the image, and the convolution layer allows the image to better fit the true value and reduce the number of channels of the image. After three convolution layers and residual blocks, the image is finally activated by the tanh function to obtain a 3-channel blurred face image restoration result.

[0089] The discriminator uses a shallow convolutional neural network as a classifier. After reducing the dimension of the input variable through the convolutional network, it makes a judgment and classifies it into two labels: "true" and "false". The network structure of the discriminator is as follows: Fig.10 shown.

[0090] In a specific embodiment, the image restoration method includes: step 1: obtaining a clear face image through a public face image data set, simulating a blur kernel through Gaussian random noise, and obtaining a blurred image through the convolution of the clear image and the blur kernel. In real life, due to the relative displacement between the photographed object and the lens during the exposure time, blurred face images are very abundant, but such images lack clear face image labels. Therefore, the clear face image is used as a label, the blur kernel is artificially synthesized, and the blurred face image is synthesized by convolution of the blur kernel and the face image; step 2: the blurred face image is input to the restoration algorithm based on the generative model proposed in the present invention; step 3: extracting image features through the convolution layer of the encoder, the encoder includes three convolution layers with a convolution kernel size of , a step size of 2, and a zero padding of 1, and each convolution layer is followed by an instance normalization layer and a linear activation function ReLU; step 4: the function input image enters the convolution residual block after passing through the convolution part. The residual block helps to achieve faster convergence and improve the reconstructed face image; Step 5: Calculate the mean and variance of the input image; Step 6: Calculate the hidden variable z through the mean and variance, the calculation method is: z = μ + σ · ε, where μ and σ represent the mean and mean square error of each sample respectively, and the random noise ε is randomly sampled from N (0, 1); Step 7: Input the hidden variable z into the decoder and perform upsampling to achieve image reconstruction; Step 8: The hidden variable doubles the size of the image through the nearest neighbor interpolation method, and then the convolution layer allows the image to better fit the true value and reduce the number of channels of the image. After three convolutional layers and residual blocks, the image is finally activated by the tanh function to obtain a 3-channel blurred face image restoration result.

[0091] like Figure 6As shown, the second aspect of the present invention provides an image restoration device 900, which includes: an acquisition module 902, used to acquire the restoration algorithm of the generative model and the first blurred face image, the restoration algorithm includes a discriminator, an encoder and a decoder; an extraction module 904, used to extract the image features of the first blurred face image through the encoder; a calculation module 906, used to calculate the latent variables of the image features through the restoration algorithm of the generative model; a restoration module 908, used to input the latent variables into the decoder, and obtain the restoration result by upsampling through the decoder; an image generation module 910, used to classify the restoration result according to the discriminator, and determine a clear face image.

[0092] Optionally, the acquisition module 902 includes an algorithm unit, which is used to set a generative adversarial network and a variational autoencoder based on process parameters of the restoration algorithm of the generative model; and determine the restoration algorithm of the generative model based on the generative adversarial network and the variational autoencoder.

[0093] like Figure 7 As shown, the third aspect of the present invention provides an electronic device 1000, including a processor 1110, a memory 1109, and a program or instruction stored in the memory 1109 and executable on the processor 1110. When the program or instruction is executed by the processor 1110, the various processes of the embodiment of the above-mentioned power balancing scheduling method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0094] Among them, the processor 1110 is used to obtain the restoration algorithm of the generative model and the first blurred face image, the restoration algorithm includes a discriminator, an encoder and a decoder; extract the image features of the first blurred face image through the encoder; calculate the latent variables of the image features through the restoration algorithm of the generative model; input the latent variables into the decoder, and obtain the restoration result by upsampling through the decoder; classify the restoration result according to the discriminator to determine a clear face image.

[0095] Optionally, processor 1110 is also used to obtain a second blurred face image, the second blurred face image includes a blur kernel; based on the process parameters of the restoration algorithm of the generative model, a generative adversarial network and a variational autoencoder are set, the variational autoencoder includes an encoder and a decoder, and the generative adversarial network includes a discriminator; wherein the discriminator uses a shallow convolutional neural network as a classifier; and the restoration algorithm of the generative model is determined based on the generative adversarial network and the variational autoencoder.

[0096] Optionally, processor 1110 is further used to obtain a convolution residual block; convolve the blurred face image downward through an encoder to reduce the image size and determine a first image; use a neural network to calculate the mean μ(I) and standard deviation σ(I) of the first image; sample the weight parameter ε from a standard normal distribution N(0, 1), and determine the latent variable z through the formula z=μ(I)+σ(I)·ε; upsample the latent variable z through a decoder and then convolve it, and input it into the convolution residual block to determine a second image; obtain a true value sample corresponding to the blurred face image, input the true value sample and the second image into a discriminator to determine a training set result; and determine a restoration algorithm for the generative model based on the training set result.

[0097] Optionally, processor 1110 is further configured to obtain a clear face image as a sample from a public face image database; simulate a blur kernel by means of Gaussian random noise; and determine a second blurred face image based on a convolution synthesis of the blur kernel and the sample.

[0098] The fourth aspect of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the embodiment of the above-mentioned power balancing scheduling method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0099] The methods may be implemented in a variety of different ways depending on the specific features and / or example applications. For example, the methods may be implemented by a combination of hardware, firmware, and / or software. For example, in a hardware implementation, the processor may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, electronic devices, other device units for performing the above functions, and / or combinations thereof.

[0100] A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above devices, but is not limited thereto. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory card, floppy disk, encoding mechanical device (such as a punch card or a groove with a raised structure with instructions recorded) and any suitable combination of the above devices. The computer readable storage medium used herein should not be understood as a transmission signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium, or an electrical signal transmitted through a wire, etc.

[0101] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0102] The fifth aspect of the present invention provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the embodiments of the above-mentioned power balance scheduling method, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0103] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although the features may work as above in certain combinations and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0104] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0105] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0106] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0107] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An image restoration method, characterized in that: include: Obtaining a restoration algorithm of a generative model and a first blurred face image, wherein the restoration algorithm of the generative model includes a discriminator, an encoder, and a decoder; Extracting image features of the first blurred face image by the encoder; Calculate the latent variables of the image features by using the restoration algorithm of the generative model; Inputting the latent variable into the decoder, and performing upsampling through the decoder to obtain a restored result; The restoration result is classified according to the discriminator to determine a clear face image.

2. The image restoration method according to claim 1, characterized in that: Before obtaining the restoration algorithm of the generative model and the first blurred face image, the method further includes: Acquire a second blurred face image, wherein the second blurred face image includes a blur kernel; Based on the process parameters of the restoration algorithm of the generative model, a generative adversarial network and a variational autoencoder are set, and the second blurred face image is transmitted to the variational autoencoder, the variational autoencoder includes the encoder and the decoder, and the generative adversarial network includes the discriminator; Wherein, the discriminator adopts a shallow convolutional neural network as a classifier; A restoration algorithm of a generative model is determined according to the manner in which the generative adversarial network and the variational autoencoder process the second blurred facial image.

3. The image restoration method according to claim 2, characterized in that: The method of determining the restoration algorithm of the generative model according to the way in which the generative adversarial network and the variational autoencoder process the second blurred face image includes: Get the convolution residual block; Down-convolving the second blurred face image through the encoder to reduce the image size, and inputting it into the convolution residual block to determine the first image; Calculating the mean μ(I) and standard deviation σ(I) of the first image using a neural network; Sample ε from the standard normal distribution N(0,1) and determine the latent variable z using the formula z=μ(I)+σ(I)·ε; The latent variable z is upsampled and convolved by the decoder, and input into the convolution residual block to determine a second image; Obtaining a true value sample corresponding to the second blurred face image, inputting the true value sample and the second image into the discriminator, and determining a training set result; A restoration algorithm of the generative model is determined according to the training set results.

4. The image restoration method according to claim 2, characterized in that: The step of obtaining a second blurred face image comprises: Obtain clear face images as samples from a public face image database; Simulate blur kernel by means of Gaussian random noise; A second blurred face image is determined based on the convolution synthesis of the blur kernel and the samples.

5. The image restoration method according to claim 3, characterized in that: The convolution residual block includes a convolution layer, an instance normalization layer and a linear activation function, and the convolution layer is set with a convolution kernel size parameter, a step parameter and a zero padding parameter.

6. The image restoration method according to claim 1, characterized in that: The encoder comprises: a convolution layer, an instance normalization layer and a linear activation function, wherein the convolution layer is provided with a convolution kernel size parameter, a step size parameter and a zero padding parameter; The structure of the decoder is the same as that of the encoder.

7. An image restoration device, characterized in that: include: An acquisition module, used to acquire a restoration algorithm of a generative model and a first blurred face image, wherein the restoration algorithm of the generative model includes a discriminator, an encoder and a decoder; An extraction module, configured to extract image features of the first blurred face image through the encoder; A calculation module, used for calculating the latent variables of the image features through the restoration algorithm of the generative model; A restoration module, used for inputting the latent variable into the decoder, and obtaining a restoration result by upsampling through the decoder; The image generation module is used to classify the restoration results according to the discriminator to determine a clear face image.

8. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image restoration method as claimed in any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the image restoration method according to any one of claims 1 to 6 are implemented.

10. A chip, characterized in that: The chip includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or an instruction to implement the steps of the image restoration method according to any one of claims 1 to 6.

Images