A face image restoration method based on autoregressive generative model

Abstract

The present invention discloses a facial image restoration method based on an autoregressive generative model, which belongs to the field of image processing technology. The present invention constructs a facial image restoration model based on mask autoregressive generation, and uses the model to achieve restoration of high-definition facial images. Compared with the facial image restoration method based on a generative adversarial network, the present invention does not introduce adversarial training during the training process, so it is not easy to produce artifacts in the restoration result, and the training process is easier; compared with the facial image restoration method based on a diffusion model, the present invention does not introduce random noise, so it is not easy to produce false texture information in the restoration result, has better restoration fidelity, and greatly improves the image inference speed. In addition, the present invention belongs to a facial image restoration method based on vector quantization, which solves the problem of accurate mapping of degraded images to high-quality image encoding under the framework of this method, and can achieve a more robust image restoration effect.

Description

Technical Field

[0001] The present invention provides a facial image restoration method, in particular to a facial image restoration method based on an autoregressive generative model, and belongs to the technical field of image processing. Background Art

[0002] Facial image restoration requires reconstructing high-quality facial details from degraded facial images or photographs. However, due to the complexity of the degradation, which can even be difficult to represent explicitly, the problem is mathematically ill-posed. To address this issue, existing facial image restoration methods typically incorporate prior information from generative models into the restoration process to simplify the problem.

[0003] Existing methods for introducing generative models in facial image restoration include those based on generative adversarial networks (GANs) and diffusion models. GAN-based methods project degraded images into the GAN's latent space, thereby leveraging the generative power of pre-trained GANs to restore realistic facial image details. Representative technologies include GPEN and GFPGAN. However, such models are difficult to train and prone to generating artifacts in images. Diffusion model-based methods have stronger generative priors and typically utilize pre-trained diffusion models to enhance degraded images. Representative technologies include DiffBIR and DR2. However, diffusion models introduce excessive randomness into the restoration process, and the inference process is very time-consuming.

[0004] To overcome the uncertainty introduced by the above methods, a facial image restoration method based on vector quantization (VQ) has been proposed. This method uses a vector quantization autoencoder to transform the image from a continuous pixel space to a less complex discrete latent space, thereby simplifying the problem to a discrete code prediction problem in the latent space. Representative techniques include VQFR and Codeformer. However, these methods do not fully consider the distribution differences between degraded and high-quality images in the latent space, and therefore still cannot effectively solve the code prediction problem in the latent space. Summary of the Invention

[0005] To address the poor restoration results and low efficiency of existing facial image restoration techniques, this paper provides a facial image restoration method based on an autoregressive generative model. This method, specifically a VQ-based facial image restoration method, focuses on accurately mapping degraded images to high-quality image codes in a latent space.

[0006] The technical solutions provided by the present invention are as follows:

[0007] A facial image restoration method based on an autoregressive generative model comprises the following steps:

[0008] A. Collecting facial image data: By combining degradation operators to simulate degradation in real scenarios, we generate one-to-one or one-to-many degraded images for each clear image to construct a training dataset.

[0009] B. Construction of face image restoration model based on autoregressive generation, the implementation method is:

[0010] B1. Construct an image quantizer to discretize the facial image data collected in step A.

[0011] B2. Construct a conditional autoregressive generative model, specifically comprising a degraded image encoder and an autoregressive generative model. The degraded image encoder converts the degraded image into degraded image features and uses them as conditions to guide autoregressive generative process. The autoregressive generative model receives two inputs: the degraded image features encoded by the degraded image encoder and the masked clear image code. The degraded image features serve as conditions to guide the autoregressive generative process and complete the masked clear image code.

[0012] B3. Learning module settings;

[0013] B4. Hyperparameter settings;

[0014] C. Use the training data obtained in step A to train the mask autoregressive face image restoration model constructed in step B. The implementation method is as follows:

[0015] C1. Parameter initialization;

[0016] C2. Forward propagation and loss function calculation;

[0017] C3. Parameter update;

[0018] C4. Repeat steps C1 to C3 until the loss function converges;

[0019] D. Based on the output of the autoregressive generative model in step B, the high-definition face image is restored through the decoder of the image quantizer. The implementation method is as follows:

[0020] D1. Generation of input for autoregressive generative models;

[0021] D2. Autoregressive generation: Specifically, the autoregressive generation model first predicts the probabilities of all code positions and uses this as the probability distribution of each code. Next, based on the probability distribution of each code, a new code is sampled from the masked code. Finally, the generated new code is fed into the masked autoregressive model as input. This process is repeated T times to ultimately generate a high-definition image code.

[0022] D3. Encode the high-definition image generated in step D2 and input it into the decoder of the image quantizer, and decode it to generate a restored image.

[0023] Furthermore, an image acquisition device is used to acquire images with portraits of people; the acquired images are registered and cropped to adapt to subsequent neural network input.

[0024] Furthermore, the degradation operator includes Gaussian noise, motion blur, JPEG compression, etc.

[0025] Furthermore, the image quantizer is a model based on a convolutional neural network or a model based on a Transformer structure.

[0026] Furthermore, the forward propagation process is carried out along two routes, specifically including:

[0027] The first route first uses the clear image code generator to generate the modeling target of the autoregressive generative model - the clear image code, and then uses the quantizer module in the image quantizer to convert the clear image code into continuous clear image features for subsequent loss function calculation between the degraded image features;

[0028] The second route encodes the clear image through the mask layer to generate a coding sequence, which is then input into the autoregressive generative model to generate the probability distribution of the subsequent sequence for the calculation of the subsequent loss function.

[0029] Furthermore, the loss function includes L1 loss, L2 loss, cross entropy loss, negative log-likelihood function, etc.

[0030] Compared with the prior art, the present invention has the following beneficial effects.

[0031] Compared with the GAN-based face image restoration method, the present invention does not introduce adversarial training in the model training process, so it is less likely to produce artifacts in the restoration results, and the training process is easier;

[0032] Compared with the facial image restoration method based on the diffusion model, the present invention does not introduce random noise, so it is less likely to produce false texture information in the restoration result, has better restoration fidelity, and greatly improves the image inference speed;

[0033] Compared with the face image restoration method based on vector quantization, the present invention focuses on solving the problem of accurate mapping of degraded images to high-quality image coding in latent space, and can achieve more robust image restoration effects, especially under severely degraded conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1This is a schematic diagram of facial image restoration based on the masked autoregressive generative model of the present invention.

[0035] Figure 2 FIG. 4 is a schematic diagram of an image quantizer according to a specific embodiment of the present invention.

[0036] Figure 3 It is a specific implementation of the present invention, where (a) represents the training method and (b) represents a single reasoning step. DETAILED DESCRIPTION

[0037] The present invention provides a method for restoring facial images based on a mask autoregressive generative model. Figure 1 As shown, the specific implementation steps include:

[0038] A. Collection of facial image data and construction of training datasets, implemented as follows:

[0039] A1. Face image collection. We use the open-source face image dataset FFHQ as the original clean images. This dataset contains 70,000 high-definition face images.

[0040] A2. Image preprocessing: Align the images in the dataset above to ensure the face is centered in the image and crop them to a uniform size of 512×512 pixels.

[0041] A3. Degraded image synthesis. The clean image is blurred, downsampled, noisy, JPEG compressed, and upsampled in sequence using the following degradation model to synthesize the degraded image.

[0042]

[0043] where σ, r, δ, and q are randomly sampled from [1, 15], [1, 30], [0, 20], and [40, 100], respectively.

[0044] B. Construction of a face image restoration model based on mask autoregression generation, the implementation method is:

[0045] B1. Construct an image quantizer to discretize the data collected in step A. This example uses a VQGAN as an image quantizer, such as Figure 2 As shown in the figure, VQGAN consists of an encoder, a quantizer, and a decoder. First, the encoder E h A clear image Encoding into feature space That is, f h =E h (I h ). The quantizer module contains a code table During the quantization process, the code table C is searched one by one with the characteristics The nearest neighbor vector is replaced to obtain the quantitative features

[0046]

[0047] Among them, q (i,j) Indicates the corresponding index position in the code table, It is the discrete representation corresponding to the clear image. In the decoder part, the quantized feature Input decoder D h The reconstructed image can be obtained Right now

[0048] B2. Construct a mask-based conditional autoregressive generative model. This step includes two modules: the degraded image encoder and the mask autoregressive model. The degraded image encoder uses the encoder part of VQGAN as the basic architecture. During the training process, the degraded image Converted into degraded image features f l This is used as a condition to guide masked autoregressive generation. The masked autoregressive model includes 12 layers of bidirectional self-attention modules, with cross-layer connections between modules to optimize feature transfer. The autoregressive model receives two inputs: degraded image features obtained by encoding the degraded image encoder and a masked high-quality image code. The degraded image features serve as a condition to guide the autoregressive generation process, completing the masked clean image code.

[0049] B3. Learning module settings. In this example, Kaiming He initialization is used as the parameter initialization module, Adam is used as the parameter update module, and linear gradient decay is used as the learning rate update module.

[0050] B4. Hyperparameter settings. In this example, the batch size is set to 24 and the initial learning rate is set to 3×10 -4 And linearly decay to 1×10 -5 , set the number of iterations to 500K.

[0051] C. Use the training data obtained in A to train the face restoration model based on mask autoregression generated in B, such as Figure 3 As shown, the implementation method is:

[0052] C1. Parameter initialization. The present invention uses the Kaiming He initialization algorithm to initialize the parameters Θ of the masked autoregressive model;

[0053] C2. Input the training data from step A into the constructed masked autoregressive model and perform forward propagation. During the training process, define the autoregressive condition extractor (including a degraded image encoder and a linear layer) and the clear image code generator (including the clear image encoder and image quantizer in the image quantizer). The autoregressive condition extractor converts the degraded image into the guidance conditions required for autoregressive generation, while the clear image code generator can convert the clear image into its corresponding discrete coding representation. The forward propagation process is carried out separately along two routes.

[0054] The first route first uses a clean image encoding generator to generate the modeling target of the mask autoregressive model. Then use the quantizer module in the image quantizer to convert the clear image code q into a clear image feature In order to calculate the subsequent feature loss;

[0055] The second route generates a coding sequence by passing the clear image code q through the mask layer M in Indicates the binary mask. In addition to the 1024 codes of the vocabulary, a special tag, [MASK], is defined. i =1 when encoding q i is replaced by [MASK], when m i =0 when encoding q i Remain unchanged. Finally, and degraded image features f l Input the prediction of the mask position encoding output in the bidirectional self-attention module In order to calculate the subsequent cross entropy loss.

[0056] C3. Loss function calculation. One of the training objectives of the present invention is to calculate the cross entropy loss between the autoregressive model output and the clean image encoding, that is, to optimize the following maximum likelihood function

[0057]

[0058] The present invention also uses the L2 loss function as the feature loss to make the feature f of the degraded image after passing through the encoder l Close to the quantized features of high-quality images Its loss function is

[0059]

[0060] Therefore, the overall loss function can be written as

[0061]

[0062] During the training process, tokenSet to 0.5;

[0063] C4. Parameter update: Use the Adam algorithm to update the model parameters Θ.

[0064] D. Based on the output of the mask autoregressive model in step B, high-definition face image restoration is achieved through the decoder part of the image quantizer.

[0065] D1. Generation of mask autoregressive model input. The degraded image is converted into degraded image features f with the help of autoregressive conditional extractor. l , take the encoding sequence of all [Mask] as the initialization encoding q (0) Next, f l With q (0) Mapped to the same dimension and input into the masked autoregressive model;

[0066] D2. Masked autoregressive generation. In each prediction step t, the masked autoregressive model first predicts the probability of all encoding positions And use it as the probability distribution of each code; then, according to Sample from the masked code codes, where γ() represents the sampling strategy. Finally, the generated new code q (t) This is fed into the mask autoregressive model as input. This process is repeated T times to generate a high-definition image code.

[0067] D3. Image decoding: The high-definition image code q generated in step D2 is input into the decoder part of the image quantizer, and decoded to produce a restored image.

[0068] Although the example scheme of the present invention has been shown, it is not limited to what is shown in the description and embodiments, and it can be fully applied to various fields suitable for the present invention. The method proposed in the present invention is not only used for the restoration of mixed degraded facial images. Various single degradation restoration tasks including but not limited to image super-resolution, deblurring, and image enhancement tasks including but not limited to completion, coloring and other applications all fall within the scope of protection of the present invention. All downstream application tasks based on the method proposed in the present invention fall within the scope of protection of the present invention, such as old photo restoration. Those skilled in the art will understand that without departing from the principles and purpose of the present invention, various changes and improvements can be made to the embodiments, which all fall within the scope of protection of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A facial image restoration method based on an autoregressive generative model, characterized in that: The steps include: A. Collecting facial image data: By combining degradation operators to simulate degradation in real scenarios, we generate one-to-one or one-to-many degraded images for each clear image to construct a training dataset. B. Construction of face image restoration model based on autoregressive generation, the implementation method is: B1. Construct an image quantizer to discretize the facial image data collected in step A. B2. Construct a conditional autoregressive generative model, specifically comprising a degraded image encoder and an autoregressive generative model. The degraded image encoder converts the degraded image into degraded image features and uses them as conditions to guide autoregressive generative process. The autoregressive generative model receives two inputs: the degraded image features encoded by the degraded image encoder and the masked clear image code. The degraded image features serve as conditions to guide the autoregressive generative process and complete the masked clear image code. B3. Learning module settings; B4. Hyperparameter settings; C. Use the training data obtained in step A to train the mask autoregressive face image restoration model constructed in step B. The implementation method is as follows: C1. Parameter initialization; C2. Forward propagation and loss function calculation; C3. Parameter update; C4. Repeat steps C1 to C3 until the loss function converges; D. Based on the output of the autoregressive generative model in step B, the high-definition face image is restored through the decoder of the image quantizer. The implementation method is as follows: D1. Generation of input for autoregressive generative models; D2. Autoregressive generation: Specifically, the autoregressive generation model first predicts the probabilities of all code positions and uses this as the probability distribution of each code. Next, based on the probability distribution of each code, a new code is sampled from the masked code. Finally, the generated new code is fed into the masked autoregressive model as input. This process is repeated T times to ultimately generate a high-definition image code. D3. Encode the high-definition image generated in step D2 and input it into the decoder of the image quantizer, and decode it to generate a restored image.

2. The facial image restoration method based on the autoregressive generative model according to claim 1, characterized in that: In step A, an image acquisition device is used to acquire an image with a portrait of a person; and the acquired image is registered and cropped to adapt to subsequent neural network input.

3. The facial image restoration method based on the autoregressive generative model according to claim 1, characterized in that: The degradation operators include Gaussian noise, motion blur, and JPEG compression.

4. The facial image restoration method based on the autoregressive generative model according to claim 1, wherein: The image quantizer is a model based on a convolutional neural network or a model based on a Transformer structure.

5. The facial image restoration method based on the autoregressive generative model according to claim 1, wherein: The forward propagation process is carried out in two ways, specifically including: The first route first uses the clear image code generator to generate the modeling target of the autoregressive generative model - the clear image code, and then uses the quantizer module in the image quantizer to convert the clear image code into continuous clear image features for subsequent loss function calculation between the degraded image features; The second route encodes the clear image through the mask layer to generate a coding sequence, which is then input into the autoregressive generative model to generate the probability distribution of the subsequent sequence for the calculation of the subsequent loss function.

6. The method for facial image restoration based on an autoregressive generative model according to claim 1, wherein: The loss function includes L1 loss, L2 loss, cross entropy loss or negative log-likelihood function.

Images