Face image restoration method based on autoregressive generative model

Abstract

The invention discloses a face image restoration method based on an autoregressive generative model, and belongs to the technical field of image processing. According to the method, the face image restoration model generated based on mask autoregression is constructed, and the high-definition face image restoration is realized by using the model. Compared with a face image restoration method based on a generative adversarial network, adversarial training is not introduced in the training process, so that artifacts are not easy to generate in a restoration result, and the training process is easier; compared with a face image restoration method based on a diffusion model, random noise is not introduced, so that false texture information is not easy to generate in a restoration result, the restoration fidelity is better, and the image reasoning speed is greatly improved. Besides, the invention belongs to a face image restoration method based on vector quantization, the problem of accurate mapping from a degraded image to high-quality image coding under the framework of the method is solved, and a more robust image restoration effect can be realized.

Description

Technical Field

[0001] The present invention provides a method for restoring a facial image, in particular to a method for restoring a facial image based on an autoregressive generation model, and belongs to the technical field of image processing. Background Art

[0002] The problem of face image restoration requires reconstructing high-quality facial details from degraded face images or photos. However, due to the complexity of the degradation form or even the inability to express it explicitly, the problem is seriously ill-posed mathematically. To solve this problem, existing face image restoration methods usually introduce prior information from the generative model into the restoration process to simplify the problem.

[0003] Existing methods for introducing generative models in face image restoration include methods based on generative adversarial networks (GAN) and diffusion models. Among them, the GAN-based method projects the degraded image into the latent space of GAN, so that realistic facial image details can be restored with the help of the generation ability of pre-trained GAN. Representative technologies include GPEN and GFPGAN. However, this type of model is difficult to train and easily generates artifacts in the image. The diffusion model-based method has a stronger generation prior. This type of method usually uses a pre-trained diffusion model to enhance the degraded image. Representative technologies include DiffBIR and DR2. However, the diffusion model will introduce too much randomness to the restoration process, and the inference process is very time-consuming.

[0004] In order to overcome the uncertainty brought by the above methods to the restoration process, a face image restoration method based on vector quantization (VQ) was proposed. This type of method converts the image from a continuous pixel space to a discrete latent space with lower complexity through a vector quantization autoencoder, thereby simplifying the problem to a discrete coding prediction problem in the latent space. Representative technologies include VQFR and Codeformer. However, this type of method does not fully consider the distribution difference between degraded images and high-quality images in the latent space, so it still cannot solve the coding prediction problem in the latent space well. Summary of the invention

[0005] In view of the problems of poor restoration effect and low efficiency in existing face image restoration technologies, the present invention provides a face image restoration method based on an autoregressive generative model. In the subdivision field, the present invention belongs to a face image restoration method based on VQ, focusing on solving the problem of accurate mapping of degraded images to high-quality image coding in latent space.

[0006] The technical solution provided by the present invention is as follows:

[0007] A method for restoring a face image based on an autoregressive generative model comprises the following steps:

[0008] A. Collection of face image data: by compounding degradation operators to simulate the degradation in real scenes, one-to-one or one-to-many degraded images are generated for each clear image to construct a training data set;

[0009] B. Construction of face image restoration model based on autoregressive generation, the implementation method is:

[0010] B1. Construct an image quantizer to discretize the face image data collected in step A;

[0011] B2. Construct a conditional autoregressive generation model, which specifically includes a degraded image encoder and an autoregressive generation model. The degraded image encoder converts the degraded image into degraded image features and uses them as conditions to guide autoregressive generation. The autoregressive generation model receives two inputs: one is the degraded image features encoded by the degraded image encoder; the other is the masked clear image code, where the degraded image features are used as conditions to guide the autoregressive generation process and complete the masked clear image code;

[0012] B3. Learning module settings;

[0013] B4. Hyperparameter settings;

[0014] C. Use the training data obtained in step A to train the face image restoration model based on mask autoregression generated in step B. The implementation method is:

[0015] C1. Parameter initialization;

[0016] C2. Forward propagation and loss function calculation;

[0017] C3. Parameter update;

[0018] C4. Repeat steps C1 to C3 until the loss function converges;

[0019] D. Based on the output of the autoregressive generative model in step B, the high-definition face image is restored through the decoder of the image quantizer. The implementation method is:

[0020] D1. Generation of input for autoregressive generative models;

[0021] D2. Autoregressive generation, specifically: the autoregressive generation model first predicts the probability of all code positions and uses it as the probability distribution of each code; then, according to the probability distribution of each code, a new code is sampled from the masked code, and finally the generated new code is sent as input to the masked autoregressive model. This process is repeated for T steps to finally generate a high-definition image code;

[0022] D3. Encode the high-definition image generated in step D2 and input it into the decoder of the image quantizer, and decode it to generate a restored image.

[0023] Furthermore, an image acquisition device is used to acquire images with portraits of people; the acquired images are registered and cropped to adapt to subsequent neural network input.

[0024] Furthermore, the degradation operator includes Gaussian noise, motion blur, JPEG compression, etc.

[0025] Furthermore, the image quantizer is a model based on a convolutional neural network or a model based on a Transformer structure.

[0026] Furthermore, the forward propagation process is carried out from two routes respectively, specifically including:

[0027] The first route first uses the clear image code generator to generate the modeling target of the autoregressive generative model - the clear image code, and then uses the quantizer module in the image quantizer to convert the clear image code into continuous clear image features for subsequent loss function calculation between the degraded image features;

[0028] The second route encodes the clear image through the mask layer to generate a coded sequence, which is then input into the autoregressive generative model to generate the probability distribution of the subsequent sequence in order to calculate the subsequent loss function.

[0029] Furthermore, the loss function includes L1 loss, L2 loss, cross entropy loss, negative log-likelihood function, etc.

[0030] Compared with the prior art, the present invention has the following beneficial effects.

[0031] Compared with the face image restoration method based on GAN, the present invention does not introduce adversarial training in the model training process, so it is not easy to produce artifacts in the restoration results, and the training process is easier;

[0032] Compared with the face image restoration method based on the diffusion model, the present invention does not introduce random noise, so it is not easy to generate false texture information in the restoration result, has better restoration fidelity, and greatly improves the image reasoning speed;

[0033] Compared with the face image restoration method based on vector quantization, the present invention focuses on solving the problem of accurate mapping of degraded images to high-quality image encoding in latent space, and can achieve more robust image restoration effects, especially under severely degraded conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1It is a schematic diagram of facial image restoration based on the mask autoregressive generative model of the present invention.

[0035] Figure 2 It is a schematic diagram of an image quantizer according to a specific embodiment of the present invention.

[0036] Figure 3 It is a specific implementation of the present invention, wherein (a) represents a training method and (b) represents a single reasoning step. DETAILED DESCRIPTION

[0037] The present invention provides a method for restoring a face image based on a mask autoregressive generative model. Figure 1 As shown, the specific implementation steps include:

[0038] A. Collection of face image data and construction of training data set, the implementation method is:

[0039] A1. Face image collection. We use the open source face image dataset FFHQ as the original clean image. The dataset contains 70,000 high-definition face images.

[0040] A2. Image preprocessing: The images in the above datasets are registered to ensure that the face is located in the center of the entire image, and are uniformly cropped to a size of 512×512 pixels;

[0041] A3. Degraded image synthesis. The clean image is blurred, downsampled, noisy, JPEG compressed, and upsampled in sequence using the following degradation model to synthesize the degraded image.

[0042]

[0043] where σ, r, δ, q are randomly sampled from [1, 15], [1, 30], [0, 20], and [40, 100] respectively.

[0044] B. Construction of face image restoration model based on mask autoregression generation, the implementation method is:

[0045] B1. Build an image quantizer to discretize the data collected in step A. This example uses a VQGAN as an image quantizer, such as Figure 2 As shown in the figure, VQGAN consists of an encoder, a quantizer, and a decoder. First, the encoder E h A clear image Encoding into feature space That is, f h =E h (I h ). The quantizer module contains a code table During the quantization process, the code table C is searched one by one with the characteristics The nearest neighbor vector is replaced to obtain the quantitative features

[0046]

[0047] Among them, q (i,j) Indicates the corresponding index position in the code table, is the discrete representation corresponding to the clear image. In the decoder part, the quantized feature Input decoder D h The reconstructed image can be obtained Right now

[0048] B2. Construct a mask-based conditional autoregressive generative model. This step includes two modules, the degraded image encoder and the mask autoregressive model. The degraded image encoder uses the encoder part of VQGAN as the basic architecture. Transformed into degraded image features f l The masked autoregressive model contains 12 layers of bidirectional self-attention mechanism modules, and cross-layer connections are added between modules to optimize the feature transfer process. The autoregressive model receives two inputs, one is the degraded image features obtained by the degraded image encoder; the other is the masked high-quality image code. The degraded image features are used as conditions to guide the autoregressive generation process and complete the masked clean image code.

[0049] B3. Learning module setting. The present invention sets Kaiming He initialization as the parameter initialization module, Adam as the parameter update module, and linear gradient decay as the learning rate update module;

[0050] B4. Hyperparameter settings. In this example, the batch size is set to 24 and the initial learning rate is set to 3×10 -4 And linearly decay to 1×10 -5 , set the number of iterations to 500K.

[0051] C. Use the training data obtained in A to train the face restoration model based on mask autoregression generated in B, such as Figure 3 As shown, the implementation method is:

[0052] C1. Parameter initialization. The present invention uses the Kaiming He initialization algorithm to initialize the parameters θ of the masked autoregressive model;

[0053] C2. Input the training data in step A into the constructed mask autoregressive model and forward propagate. During the training process, define the autoregressive condition extractor (including a degraded image encoder and a linear layer) and the clear image code generator (including the clear image encoder and image quantizer in the image quantizer). The autoregressive condition extractor converts the degraded image into the guiding conditions required for autoregressive generation, while the clear image code generator can convert the clear image into its corresponding discrete coding representation. The forward propagation process is carried out from two routes respectively.

[0054] The first route first uses a clean image encoding generator to generate the modeling target of the mask autoregressive model. Then use the quantizer module in the image quantizer to convert the clear image code q into a clear image feature In order to calculate the subsequent feature loss;

[0055] The second route generates a coding sequence by passing the clear image code q through the mask layer M in Indicates the binary mask. In addition to the 1024 codes of the vocabulary, a special tag, [MASK], is defined. i =1 when encoding q i is replaced by [MASK], when m i = 0 when encoding q i Remain unchanged. Finally and the degraded image feature f l Input bidirectional self-attention module to output prediction of mask position encoding In order to calculate the subsequent cross entropy loss.

[0056] C3. Loss function calculation. One of the training objectives of the present invention is to calculate the cross entropy loss between the autoregressive model output and the clean image encoding, that is, to optimize the following maximum likelihood function

[0057]

[0058] The present invention also uses the L2 loss function as the feature loss to make the feature f of the degraded image after passing through the encoder l Close to the quantized features of high-quality images Its loss function is

[0059]

[0060] Therefore, the overall loss function can be written as

[0061]

[0062] During the training process, tokenSet to 0.5;

[0063] C4. Parameter update. Use the Adam algorithm to update the model parameters Θ.

[0064] D. Based on the output of the mask autoregressive model in step B, the high-definition face image is restored through the decoder part of the image quantizer.

[0065] D1. Generation of masked autoregressive model input. The degraded image is converted into degraded image features f with the help of autoregressive conditional extractor. l , take the encoding sequence of all [Mask] as the initialization encoding q (0) Next, f l With q (0) Mapped to the same dimension and input into the masked autoregressive model;

[0066] D2. Masked autoregressive generation. In each prediction step t, the masked autoregressive model first predicts the probabilities of all encoding positions And use it as the probability distribution of each code; then, according to Sample from the masked code codes, where γ() represents the sampling strategy. Finally, the generated new code q (t) The input is sent to the mask autoregressive model. This process is repeated T steps to finally generate a high-definition image code;

[0067] D3. Image decoding: The high-definition image code q generated in step D2 is input into the decoder part of the image quantizer, and decoded to generate a restored image.

[0068] Although the example scheme of the present invention has been shown, it is not limited to that shown in the specification and embodiments, and it can be fully applied to various fields suitable for the present invention. The method proposed in the present invention is not only used for restoration of mixed degraded face images. Various single degradation restoration tasks including but not limited to image super-resolution, deblurring, and image enhancement tasks including but not limited to completion, coloring and other applications all belong to the protection scope of the present invention. All downstream application tasks based on the method proposed in the present invention belong to the protection scope of the present invention, such as old photo restoration. Those of ordinary skill in the art can understand that: without departing from the principles and purpose of the present invention, various changes and improvements can be made to the embodiments, which all belong to the protection scope of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A facial image restoration method based on an autoregressive generative model, characterized in that: The steps include: A. Collection of face image data: by compounding degradation operators to simulate the degradation in real scenes, one-to-one or one-to-many degraded images are generated for each clear image to construct a training data set; B. Construction of face image restoration model based on autoregression generation, the implementation method is: B1. Construct an image quantizer to discretize the face image data collected in step A; B2. Construct a conditional autoregressive generation model, which specifically includes a degraded image encoder and an autoregressive generation model. The degraded image encoder converts the degraded image into degraded image features and uses them as conditions to guide autoregressive generation. The autoregressive generation model receives two inputs: one is the degraded image features encoded by the degraded image encoder; the other is the masked clear image code, where the degraded image features are used as conditions to guide the autoregressive generation process and complete the masked clear image code; B3. Learning module settings; B4. Hyperparameter settings; C. Use the training data obtained in step A to train the face image restoration model based on mask autoregression generated in step B. The implementation method is: C1. Parameter initialization; C2. Forward propagation and loss function calculation; C3. Parameter update; C4. Repeat steps C1 to C3 until the loss function converges; D. Based on the output of the autoregressive generative model in step B, the high-definition face image is restored through the decoder of the image quantizer. The implementation method is: D1. Generation of input for autoregressive generative models; D2. Autoregressive generation, specifically: the autoregressive generation model first predicts the probability of all code positions and uses it as the probability distribution of each code; then, according to the probability distribution of each code, a new code is sampled from the masked code, and finally the generated new code is sent as input to the masked autoregressive model. This process is repeated for T steps to finally generate a high-definition image code; D3. Encode the high-definition image generated in step D2 and input it into the decoder of the image quantizer, and decode it to generate a restored image.

2. The method for restoring a facial image based on an autoregressive generative model as claimed in claim 1, characterized in that: In the step A, an image acquisition device is used to acquire images with portraits of people; and the acquired images are registered and cropped to adapt to subsequent neural network input.

3. The method for restoring a facial image based on an autoregressive generative model as claimed in claim 1, characterized in that: The degradation operators include Gaussian noise, motion blur, and JPEG compression.

4. The method for restoring a facial image based on an autoregressive generative model as claimed in claim 1, characterized in that: The image quantizer is a model based on a convolutional neural network or a model based on a Transformer structure.

5. The method for restoring a facial image based on an autoregressive generative model as claimed in claim 1, characterized in that: The forward propagation process is carried out from two routes respectively, specifically including: The first route first uses the clear image code generator to generate the modeling target of the autoregressive generative model - the clear image code, and then uses the quantizer module in the image quantizer to convert the clear image code into continuous clear image features for subsequent loss function calculation between the degraded image features; The second route encodes the clear image through the mask layer to generate a coded sequence, which is then input into the autoregressive generative model to generate the probability distribution of the subsequent sequence in order to calculate the subsequent loss function.

6. The method for restoring a facial image based on an autoregressive generative model as claimed in claim 1, characterized in that: The loss function includes L1 loss, L2 loss, cross entropy loss or negative log-likelihood function.

Images