Face image restoration method and system based on diffusion model
Through the face image recovery method based on the diffusion model, image recovery is performed using the identity information of the reference face image, which solves the problem of poor recovery of blind face images in the prior art, and achieves high-quality and identity consistency image recovery Effect.
Patent Information
- Application Number
- CN202411130670.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-06
AI Technical Summary
The existing face image recovery method is not effective when processing blind face images of unknown degradation types, and it is difficult to ensure that the character identities in the recovery results are consistent with the real identity. Especially when the real image degradation is inconsistent with the training model, the recovery effect is seriously damaged. .
A face image recovery method based on the diffusion model is adopted, and the low-frequency information of the to-processed image is extracted and the training diffusion model is used for recovery, and combined with the character identity information in the reference face image as a guide to ensure the identity consistency of the recovery result. The specific steps include building a training data set, extracting the identity information encoding vector, generating negative prompt images, performing model training and image recovery, gradually restoring high-frequency details, and ensuring the clarity and identity matching of the restored image.
It significantly improves the accuracy and accuracy of facial image recovery, ensures that the character identity of the restored image is consistent with the real character identity, improves the image recovery quality, and shows excellent performance especially in severe degradation.
Smart Images

Figure CN119090778A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image restoration, and in particular relates to a method and system for restoring a face image based on a diffusion model. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] Face restoration aims to restore high-quality, high-resolution images from low-quality face images. This technology effectively repairs damage caused by image compression, noise, blur and other degradation factors by improving the clarity, color, contrast and other aspects of the image. On the one hand, face image restoration plays a vital role in multiple downstream tasks, including video surveillance, face recognition, face detection, etc., because a clear face image can significantly improve the accuracy of facial recognition algorithms; on the other hand, in real life, mobile imaging systems have great limitations, and face photos taken at medium and long distances are often not clear enough. The method of improving the quality of photos by upgrading mobile imaging hardware usually requires a lot of manpower and financial resources. In contrast, the use of face restoration algorithms can not only effectively improve the quality of captured images, but also be low-cost and widely applicable.
[0004] In practical applications, non-blind face image restoration faces significant limitations, especially when the real image degradation is inconsistent with the degradation model used in training, the restoration effect will be seriously impaired. This situation highlights the necessity of blind face image restoration, that is, to develop a face restoration technology that can adaptively recognize and handle unknown degradation types.
[0005] The core goal of blind face image restoration is to restore low-quality face images affected by multiple complex degradation factors such as noise, blur and downsampling to high-definition face images. This process involves adding high-frequency details to the low-frequency areas of the image. As the image degradation deepens, the required supplementary details also increase accordingly. In essence, the process of supplementing details can be regarded as image generation. Therefore, the image restoration task is similar to the image generation task to a certain extent. The blind face image restoration task is also a highly pathological task, because a degraded face image often corresponds to multiple feasible high-quality face images. Current face image restoration methods usually tend to produce moderate solutions and cannot guarantee that the identity of the person in the face image restoration result is consistent with the real person's identity. Summary of the invention
[0006] In order to solve the above problems, the present invention proposes a facial image restoration method and system based on a diffusion model. The present invention utilizes a diffusion model to restore blind facial images, and uses the character information in a reference facial image as a guide, so that the character identity information of the restored facial image is consistent with the real character identity information.
[0007] According to some embodiments, the present invention adopts the following technical solutions:
[0008] A method for restoring a face image based on a diffusion model comprises the following steps:
[0009] Extract low-frequency information from the face image to be processed;
[0010] The extracted low-frequency information is used as input and the trained diffusion model is used to perform image restoration. During this process, the person identity information in the reference face image is used as a guide to ensure that the restored face image matches the identity information of the real person.
[0011] As an optional implementation, the following steps are also included:
[0012] Obtain a face image to be processed and a reference face image.
[0013] As an optional implementation, the person identity information in the reference face image is extracted by a face recognition module and encoded into an identity information encoding vector.
[0014] As a further implementation, the identity information encoding vector is used to guide the training and recovery process of the diffusion model so that the restored face image keeps matching the identity of the person in the reference face image.
[0015] As an optional implementation, the training process of the diffusion model includes:
[0016] Build a training dataset;
[0017] The image pairs in the training data set are used as training data, and the identity information encoding vectors of the image pairs are extracted to guide the diffusion model to be trained;
[0018] Generate a description of the reference face image and completely negate the adjectives in the description to generate a negative prompt;
[0019] According to the negative hint, an image consistent with the description is generated, and a negative reference face image is obtained through face detection and cropping;
[0020] A small amount of degradation is performed on the negative reference face images to avoid the attributes of the negative cue words appearing in the images generated by the diffusion model.
[0021] As a further limited implementation, the process of constructing a training data set includes: acquiring a plurality of face images that meet quality requirements to form a data set, wherein the face images cover different postures, expressions and / or shooting scenes;
[0022] The pitch angle and yaw angle are transformed to generate multiple images with different postures;
[0023] The face image without any emotion or expression in each posture is used as the posture reference image;
[0024] Combine each image in the dataset with each pose reference image to obtain an image pair;
[0025] Extracting attributes from each image of each image pair, and replacing the pose and expression attributes in the pose reference image to generate a new 3D pose image;
[0026] Calculate the identity similarity between each image in the data set and its corresponding generated multiple images with different postures, and filter out images with identity similarity greater than the set value;
[0027] Generate the final training dataset, which contains all the images of the original dataset and the selected synthetic reference images.
[0028] As an optional implementation method, the specific process of image restoration using the trained diffusion model includes: the diffusion model takes low-frequency information as input and gradually restores the high-frequency details in the image. In each iteration, the identity information encoding vector and the negative prompt image are used as control signals to guide the restoration process, and finally a face image that meets the clarity is generated, and the person identity information in the restored image is consistent with the person identity information of the reference face image.
[0029] A face image restoration system based on a diffusion model, comprising:
[0030] An extraction module is configured to extract low-frequency information from the face image to be processed;
[0031] The face image restoration module is configured to take the extracted low-frequency information as input and use the trained diffusion model to perform image restoration. During this process, the person identity information in the reference face image is used as a guide to ensure that the restored face image matches the identity information of the real person.
[0032] A computer program product comprises a computer program, and when the computer program is executed by a processor, the steps in the above method are implemented.
[0033] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps in the above method are completed; or the above computer program product is stored.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] The present invention innovatively proposes a facial image restoration method, which is based on a diffusion model and uses the person identity information in a reference facial image as a guide for facial image restoration. This method can make the person identity information of the restored facial image consistent with the real person identity information, thereby improving the accuracy and precision of image restoration and greatly improving the image restoration quality.
[0036] The present invention innovatively proposes a method for constructing a training data set that can support the training of a diffusion model. First, the pitch angle and yaw angle are set, and a variety of different postures are obtained through permutations and combinations. The corresponding posture reference images are selected, and image pairs are constructed. The attributes of the images in the image pairs are extracted, and the images are replaced. The synthetic reference images are screened using identity similarity, and the screened synthetic reference images and the original images are jointly constructed into the final training data set, which can effectively increase the amount of training data while also ensuring the quality of the training data.
[0037] The present invention innovatively proposes a method for encoding person identity information suitable for image restoration, generates a detailed description of a reference face image, generates negative prompts based on the description, and then obtains a negative reference face image to increase diversity and avoid the appearance of attributes in negative prompt words in the image generated by the diffusion model, thereby realizing face image restoration based on the diffusion model with the person identity information in the reference face image as a guide.
[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0040] Figure 1 is a schematic diagram of a latent space diffusion model framework of an embodiment;
[0041] Figure 2 is a schematic diagram of the overall process of an embodiment;
[0042] Figure 3is a data set synthesis result of an embodiment, wherein the left side is a re-rendered posture control image, and the right side is a schematic diagram of a synthesized reference face image. DETAILED DESCRIPTION
[0043] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0044] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0045] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0046] In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.
[0047] Embodiment 1
[0048] A face image restoration method based on diffusion model, such as Figure 2 As shown, the following steps are included:
[0049] Obtaining a face image to be processed and a reference face image;
[0050] Extract low-frequency information from the face image to be processed;
[0051] The extracted low-frequency information is used as input and the trained diffusion model is used to perform image restoration. During this process, the person identity information in the reference face image is used as a guide to ensure that the restored face image matches the identity information of the real person.
[0052] Before training the diffusion model, a data set needs to be constructed. The specific process includes:
[0053] (1) Dataset preparation:
[0054] (1-1) Use the FFHQ dataset, which contains 70,000 high-quality face images covering a variety of postures, expressions, and shooting scenes;
[0055] (1-2) Set the pitch angle to {-45°, 0°, 45°} and the yaw angle to {-45°, 0°, 45°}, and get 9 different postures through permutations and combinations.
[0056] (2) Pose reference image generation: For each pose, a face image without any emotion or expression is selected as the pose reference image, denoted as {ref_j|j=1,2,...,9};
[0057] (3) Synthetic image generation:
[0058] (3-1) Combine each image x in the FFHQ dataset with each pose reference image ref_j to obtain an image pair (x, ref_j);
[0059] (3-2) Use the Arc2Face method to extract attributes from images x and ref_j, and replace the posture and expression attributes in ref_j to generate a new 3D posture image r_j.
[0060] In the subsequent process, the image x is used to provide the identity information of the person, and the 3D posture image r_j is used to control the posture of the generated image.
[0061] (4) Synthetic image screening:
[0062] (4-1) The ArcFace method is used to calculate the identity similarity between each image in the FFHQ dataset and its corresponding 9 generated images, and the generated images with identity similarity greater than 0.7 are selected as the synthetic reference images;
[0063] (4-2) Finally, the FFHQRef dataset is generated, which contains 70,000 FFHQ dataset images and 384,451 synthetic reference images.
[0064] In some other embodiments, based on the FFHQ dataset containing 70,000 high-quality face images, the Arc2Face method is used to synthesize the FFHQRef dataset that meets the above properties to make up for the deficiencies in the existing datasets. Specifically, this paper first sets the pitch angle to {-45°, 0°, -45°} and the yaw angle to {-45°, 0°, -45°}, and obtains 9 different postures by permuting and combining Pitch and Yaw. Based on the posture setting, a face image without any emotion or expression is selected for each posture as a posture reference image, denoted as {refj|=1,2,...,9}, and then these 9 posture reference images are combined with each image in the FFHQ dataset to obtain {(x,refj)|x∈FFHQ,j=1,2,...,9}. Following the Arc2Face method, EMOCA is used to extract the attributes of x and refj, respectively, denoted as:
[0065] Ec(x)→(β,θ,ψ,α,l,δ,c)Ec(refj)→(βj,θj,ψj,αj,lj,δj,cj)
[0066] Among them, β represents shape code, θ represents pose code, ψ represents expression code, α represents albedo code, l represents light code, δ represents detail code, and c represents camera code. Substituting βj and δj attributes in refj with β and δ attributes of x, we get Condj = (β, θj, ψj, αj, lj, δ, cj). The 3D pose image rj can be re-rendered through the Condj attribute set.
[0067] Combining Arc2Face and ControlNet, using x as the input of arc2face to provide character identity information, and rj as the input of Controlnet to control the posture of the generated image, we finally obtained 630,000 generated images.
[0068] In order to improve the quality of synthesized images, the ArcFace method is used to calculate the identity similarity between each image in the FFHQ dataset and the corresponding 9 generated images, and the generated images with identity similarity greater than 0.7 are screened out. The FFHQRef dataset finally contains 70,000 FFHQ dataset images and 384,451 synthetic reference images.
[0069] After constructing the training data set, the diffusion model can be trained.
[0070] like Figure 1 As shown, this embodiment selects a latent space diffusion model, and the network is pre-trained on a large-scale data set to accumulate rich prior knowledge.
[0071] During the training process, the image pairs (x, ref_j) in the FFHQRef dataset are used as training data.
[0072] The identity information encoding vectors of x and ref_j are extracted to guide the diffusion model to train so that the restored face image is consistent with the identity of the person in the reference image.
[0073] Use the LLaVA model to generate a detailed description caption_ref for the reference face image, and use the large language model to completely negate the adjectives in caption_ref to generate a negative prompt caption_neg;
[0074] Input caption_neg into the SDXL text graph model to generate an image image_neg that is consistent with the description, and obtain the negative reference face image face_neg through face detection and cropping.
[0075] In this embodiment, images are used as negative prompt words to achieve more precise control and improve image restoration quality.
[0076] Specifically, let faceref represent the input reference face image, first use LLaVA to obtain a detailed description of the reference face image, denoted as captionref, and then use the large language model to completely negate the adjectives in captionref to obtain captionneg. For example, when captionref = <a photo of an elderly man with manymustaches and wrinkles.> When captionneg= <a photo of a young man without abeard or wrinkles.>is a feasible negative reminder.
[0077] Send captionneg to the SDXL text image model to get an image imageneg that is consistent with the description of captionneg. Then use the face detection algorithm to identify and crop the face area in imageneg to get faceneg. faceneg is a face image with completely opposite attributes to those in captionref. Using faceneg as a negative reference image can avoid the attributes in faceneg appearing in the image restored by the diffusion model, making the person information in the restored face image closer to faceref.
[0078] Some embodiments also involve training data expansion:
[0079] A small amount of degradation is performed on the negative reference face image face_neg to increase diversity and avoid the appearance of attributes in the negative prompt words in the images generated by the diffusion model.
[0080] In this embodiment, the CLIP image encoder can capture the degradation information in the reference image to a certain extent.
[0081] In practical applications, the trained diffusion model can be used for image restoration.
[0082] Low-frequency information and high-frequency details are extracted from low-quality face images. The low-frequency information is used as the input of the diffusion model, and the diffusion model gradually restores the high-frequency details in the image. In each iteration, the identity information encoding vector and the negative prompt image are used as control signals to guide the restoration process, and finally a high-definition face image is generated. The identity information of the person in the restored image is consistent with the identity information in the reference image, with rich details and high quality.
[0083] In order to verify the effect of the above embodiment, this embodiment is experimentally verified.
[0084] First, we set up the experiment. In this example, we select the CelebA-HQ and FFHQ datasets as test datasets to compare the restoration effects of different methods.
[0085] Peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and identity similarity (IS) are used as evaluation metrics.
[0086] Comparative experiments show that compared with the traditional generative adversarial network (GAN) method, the method of the present invention has significant improvements in PSNR, SSIM and IS indicators. The restored image shows excellent performance in facial details, texture and identity consistency, especially on severely degraded images.
[0087] In summary, the present invention demonstrates the effectiveness of the face image restoration method based on the diffusion model, and conducts detailed experimental verification on different datasets, proving the significant advantages of the method of the present invention in terms of image restoration quality and identity consistency.
[0088] Embodiment 2
[0089] A face image restoration system based on a diffusion model, comprising:
[0090] An image acquisition module is configured to acquire a face image to be processed and a reference face image;
[0091] An extraction module is configured to extract low-frequency information from the face image to be processed;
[0092] The face image restoration module is configured to take the extracted low-frequency information as input and use the trained diffusion model to perform image restoration. During this process, the person identity information in the reference face image is used as a guide to ensure that the restored face image matches the identity information of the real person.
[0093] Embodiment 3
[0094] A computer program product includes a computer program. When the computer program is executed by a processor, the steps in the method provided in embodiment 1 are implemented.
[0095] Embodiment 4
[0096] An electronic device, comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps in the method provided in the first embodiment are completed;
[0097] Or the computer program product described in Embodiment 3 is stored therein.
[0098] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0099] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.
Claims
1. A face image restoration method based on diffusion model, It is characterized in that The following steps are involved: Extract low-frequency information from the face image to be processed; The extracted low-frequency information is used as input and the trained diffusion model is used to perform image restoration. During this process, the person identity information in the reference face image is used as a guide to ensure that the restored face image matches the identity information of the real person.
2. A facial image restoration method based on a diffusion model as claimed in claim 1, It is characterized in that The following steps are also included: Obtain a face image to be processed and a reference face image.
3. A facial image restoration method based on a diffusion model as claimed in claim 1, It is characterized in that The person identity information in the reference face image is extracted by a face recognition module and encoded into an identity information encoding vector.
4. A facial image restoration method based on a diffusion model as claimed in claim 1 or 3, It is characterized in that The identity information encoding vector is used to guide the training and recovery process of the diffusion model so that the restored face image keeps matching the identity of the person in the reference face image.
5. The method for restoring a facial image based on a diffusion model as claimed in claim 1, It is characterized in that The training process of the diffusion model includes: Build a training dataset; The image pairs in the training data set are used as training data, and the identity information encoding vectors of the image pairs are extracted to guide the diffusion model to be trained; Generate a description of the reference face image and completely negate the adjectives in the description to generate a negative prompt; According to the negative hint, an image consistent with the description is generated, and a negative reference face image is obtained through face detection and cropping; A small amount of degradation is performed on the negative reference face images to avoid the attributes of the negative cue words appearing in the images generated by the diffusion model.
6. A facial image restoration method based on a diffusion model as claimed in claim 5, It is characterized in that The process of constructing a training data set includes: obtaining a plurality of face images that meet quality requirements to form a data set, wherein the face images cover different postures, expressions and / or shooting scenes; The pitch angle and yaw angle are transformed to generate multiple images with different postures; The face image without any emotion or expression in each posture is used as the posture reference image; Combine each image in the dataset with each pose reference image to obtain an image pair; Extracting attributes from each image of each image pair, and replacing the pose and expression attributes in the pose reference image to generate a new 3D pose image; Calculate the identity similarity between each image in the data set and its corresponding generated multiple images with different postures, and filter out images with identity similarity greater than the set value; Generate the final training dataset, which contains all the images in the original dataset and the selected synthetic reference images.
7. The method for restoring a facial image based on a diffusion model as claimed in claim 1, It is characterized in that The specific process of image restoration using the trained diffusion model includes: the diffusion model takes low-frequency information as input and gradually restores the high-frequency details in the image. In each iteration, the identity information encoding vector and the negative prompt image are used as control signals to guide the restoration process, and finally a face image that meets the clarity is generated. The identity information of the person in the restored image is consistent with the identity information of the reference face image.
8. A face image restoration system based on diffusion model, It is characterized in that include: An extraction module is configured to extract low-frequency information from the face image to be processed; The face image restoration module is configured to take the extracted low-frequency information as input and use the trained diffusion model to perform image restoration. During this process, the person identity information in the reference face image is used as a guide to ensure that the restored face image matches the identity information of the real person.
9. A computer program product, It is characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.
10. An electronic device, It is characterized in that The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of the method according to any one of claims 1 to 7 are completed; Or storing the computer program product according to claim 9.
Citation Information
Cited By
Face multi-pose generation method and system, electronic equipment and storage medium
CN120452046A
Face image reconstruction method based on ID preservation
CN120580156A
Image restoration method and device, electronic equipment and storage medium
CN121563848A
