Personalized face image restoration method and device, equipment, medium and program product

Abstract

The invention provides a personalized face image restoration method, device and equipment, a medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a low-quality image and a reference image; inputting the low-quality image and the reference image into a face image restoration model for restoration processing, and outputting a personalized face restoration image; the face image restoration model is obtained by performing personalized training on a pre-trained diffusion model based on a low-quality image sample and a reference image sample; the face restoration image is represented as a personalized image with the same face structure as the low-quality image and the same face identity as the reference image. According to the embodiment of the invention, the defect of insufficient restoration precision in the prior art is overcome, general structure fidelity and identity detail control are considered through the trained face image restoration model design, and the method is suitable for the restoration task of the seriously degraded image and has high generalization ability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a personalized facial image restoration method, apparatus, device, medium, and program product. Background Technology

[0002] Face restoration aims to recover HQ (high-quality) face images from degraded observations such as blurriness, low resolution, noise, and compression artifacts. In the real world, this task is even more challenging due to more complex degradations and variations in lighting and pose. Face restoration is a highly ill-posed problem with multiple solutions for a given LQ (low-quality) input. Compared to natural images, humans are extremely sensitive to subtle differences in face images. Even minute variations in the shape, size, or color of the eyes, nose, lips, etc., can alter a person's identity. Furthermore, we are more likely to detect subtle differences if we are familiar with a particular person's identity. Therefore, restoration models need to output not only realistic images but also images that faithfully represent the person's identity.

[0003] In related technologies, state-of-the-art face restoration methods involve generative priors, where the model is trained generatively before restoration. GFP-GAN uses a generator from StyleGAN2, which has been trained on face images to generate high-quality features. Recently, methods such as VQFR and CodeFormer have been "storing" high-quality data using discrete codebases with vector quantization and adversarial training. Some recent methods use diffusion models for restoration. DifFace uses an arbitrary restoration model to transform the LQ input into a manifold of a high-quality image, then performs forward and backward diffusion to introduce fine-grained details. DR2 uses a similar approach, first performing forward diffusion, then backward diffusion guided by a matching step in the forward diffusion; this work encourages diversity rather than personalization.

[0004] Reference-based face restoration methods utilize reference images from the same identity during the restoration process. GFRNet uses a single reference image and learns the distortion between the LQ and the reference image. ASFFNet selects the most similar reference image to reduce inaccuracies and uses adaptive feature fusion for restoration. DMDNet constructs a deep feature dictionary from important cropping regions (such as eyes, nose, and mouth). An alignment module is then used to align the features of the input and reference images, thereby fusing the features into the output image.

[0005] Current facial image restoration technologies suffer from at least the following technical shortcomings: Generative prior methods often produce unrealistic outputs due to the lack of crucial contextual information about the identity. Furthermore, existing technologies like DreamBooth require a large number of high-quality face images for training and are prone to overfitting the training data, causing the model to rely excessively on training samples during face restoration and affecting generalization ability. Additionally, this model is sensitive to noise in the input image, potentially leading to suboptimal restoration results. Perfusion may lack sufficient restoration accuracy when handling complex or severely damaged face images. It may also exhibit instability when faced with unseen types of damage or diverse facial features.

[0006] Among related technologies, ViCo relies on contextual information in the image for restoration; if the contextual information is insufficient or severely damaged, the restoration effect may be unsatisfactory. RealFill struggles to perform personalized restoration for the facial features of specific individuals. MyStyle is primarily optimized for specific individuals and is difficult to generalize to other individuals or diverse scenarios. IdentityEncoder may have insufficient accuracy when extracting facial identity features.

[0007] In related technologies, GFRNet requires a frontally high-quality (HQ) reference image with the same identity as the low-quality (LQ) image. This requirement limits its applicability in certain specific scenarios. While ASFFNet may perform well on specific datasets, its generalization ability may be limited when faced with images of different degradation types or degrees. Although DMDNet simplifies complexity through end-to-end training, it still requires significant computational resources for training and inference due to its deep learning-based nature.

[0008] Therefore, how to accurately repair low-quality facial images with different degrees of degradation is a technical problem that urgently needs to be solved. Summary of the Invention

[0009] This invention provides a personalized face image restoration method, apparatus, device, medium, and program product to address the shortcomings of existing technologies, such as insufficient restoration accuracy and low generalization ability for restoring low-quality face images with different degrees of degradation. It achieves a balance between general structural fidelity and identity detail control through a well-trained face image restoration model, making it suitable for restoration tasks of severely degraded images with high generalization ability.

[0010] In a first aspect, the present invention provides a personalized face image restoration method, comprising the following steps: Acquire low-quality images and reference images; The low-quality image and the reference image are input into the face image restoration model for restoration processing, and a personalized face restoration image is output. The face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples; the restored face image is characterized as a personalized image with the same face structure as the low-quality image and the same face identity as the reference image.

[0011] Preferably, in the personalized face image restoration method provided by the present invention, the diffusion model includes at least: a basic model block, a personalized module, and a decoder; The face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples, including: Multi-scale reference features of the reference image samples are extracted, structural features of the low-quality image samples are extracted, and initial noise processing is performed on the low-quality image samples to obtain an initial noise image. Acquire real face image samples and add noise to the real face image samples to generate real noise images at different time steps; The real noise image and the conditional vector are input into the base model block, and the intermediate features of the image are output; wherein, the conditional vector is generated based on the structural features and is used to provide multimodal conditional signals for the diffusion model; The image intermediate features and the multi-scale reference features are input into the personalization module for personalized modulation processing, and personalized features are output. Based on the real noise image and the personalized features, the predicted noise is determined. The initial noisy image and the predicted noise are input into the decoder for decoding processing, and a face restoration image sample is output. Based on the total loss function of the face image restoration model, iterative training is performed by inputting the reference image sample and the low-quality image sample into the diffusion model and outputting the face restoration image sample model until a preset termination condition is met, thus obtaining the trained face image restoration model; wherein, meeting the preset termination condition is characterized by the current iteration number reaching a preset number and / or the value of the total loss function being minimized.

[0012] Preferably, according to the personalized face image restoration method provided by the present invention, the step of inputting the intermediate features of the image and the multi-scale reference features into the personalization module for personalized modulation processing and outputting personalized features includes: Get embedded text hints; Calculate the text attention of the text embedding cue and the intermediate features of the image, and calculate the image attention of the multi-scale reference features and the intermediate features of the image; Controllable modulation features are obtained by performing point-by-point multiplication of the learnable adapter vector, the text attention, the image attention, and the intermediate features of the image. The personalized features are obtained by adding the controllable modulation features and the intermediate features of the image point by point.

[0013] Preferably, in a personalized face image restoration method provided by the present invention, the conditional vector is generated based on the structural features and includes: Cross-layer attention computation is performed on the multi-scale reference features to generate a unified identity embedding; The unified identity embedding, the structural features, and the text embedding prompts are concatenated to generate the conditional vector.

[0014] Preferably, in the personalized face image restoration method provided by the present invention, the formula for the total loss function for training the face image restoration model is: In the formula, Represented as the total loss function, This represents the improved diffusion model loss after adding low-quality image samples and reference image samples. This indicates the weights that control the generated items. Represented as regularization loss, This is represented as a cross-attention graph used to standardize learnable text embedding cue tags.

[0015] Preferably, in a personalized face image restoration method provided by the present invention, the extraction of multi-scale reference features from reference image samples includes: Acquire multiple reference image samples; The multiple reference image samples are input into the encoder of the pre-trained diffusion model for encoding processing to obtain the multi-scale reference features.

[0016] Secondly, the present invention also provides a personalized face image restoration device, comprising the following modules: The acquisition module is used to acquire low-quality images and reference images; The repair module is used to input the low-quality image and the reference image into a face image repair model for repair processing, and output a personalized face repair image; wherein, the face image repair model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples; the face repair image is characterized as a personalized image with the same face structure as the low-quality image and the same face identity as the reference image.

[0017] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the personalized face image restoration method as described above.

[0018] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the personalized face image restoration method as described above.

[0019] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the personalized face image restoration method as described above.

[0020] This invention provides a personalized face image restoration method, apparatus, device, medium, and program product. It acquires a low-quality image and a reference image; inputs the low-quality image and the reference image into a face image restoration model for restoration processing, and outputs a personalized restored face image. The face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples. The restored face image is characterized as a personalized image with the same facial structure as the low-quality image and the same facial identity as the reference image. This invention addresses the shortcomings of existing technologies, such as insufficient restoration accuracy and low generalization ability for restoring low-quality face images with varying degrees of degradation. It achieves a balance between general structural fidelity and identity detail control through a well-trained face image restoration model design, making it suitable for restoration tasks of severely degraded images with high generalization ability. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is one of the flowcharts illustrating the personalized face image restoration method provided by the present invention.

[0023] Figure 2 This is the second schematic diagram of the personalized face image restoration method provided by the present invention.

[0024] Figure 3 This is a schematic diagram of the personalized face image restoration device provided by the present invention.

[0025] Figure 4This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] The following is combined Figures 1-4 This invention describes a personalized face image restoration method, apparatus, device, medium, and program product, which addresses the shortcomings of existing technologies, such as insufficient restoration accuracy and low generalization ability for restoring low-quality face images with different degrees of degradation. It achieves a balance between general structural fidelity and identity detail control through a well-trained face image restoration model, making it suitable for restoration tasks of severely degraded images with high generalization ability.

[0028] Figure 1 This is one of the flowcharts illustrating a personalized face image restoration method provided by the present invention, such as... Figure 1 As shown, the method may include, but is not limited to, steps S100 to S200: S100, acquire low-quality image and reference image; S200, the low-quality image and the reference image are input into the face image restoration model for restoration processing, and a personalized face restoration image is output; The face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples; the restored face image is characterized as a personalized image with the same face structure as the low-quality image and the same face identity as the reference image.

[0029] In step S100 of some embodiments, a low-quality image and a reference image are acquired.

[0030] Low-quality images refer to facial images characterized by blurriness, low resolution, noise, and compression artifacts.

[0031] Reference images refer to facial images of different people with the same identity information, but different facial expressions, postures, and actions.

[0032] In step S200 of some embodiments, the low-quality image and the reference image are input into a face image restoration model for restoration processing, and a personalized face restoration image is output. The face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples; the restored face image is characterized as a personalized image with the same face structure as the low-quality image and the same face identity as the reference image.

[0033] Understandably, this invention uses a diffusion model of a basic face restoration model as its foundation. While the diffusion model can achieve realistic output, fidelity may be affected due to the ill-posed nature of the task. The strategy of this invention is to perform personalized restoration by fine-tuning the diffusion model of the basic face restoration model using some HQ reference images. To avoid catastrophic forgetting during personalized fine-tuning, this invention uses an adapter to maintain the integrity of the prior. The existing prior is preserved by freezing the diffusion model of the basic face restoration model and training only the adapter block. To avoid rapid changes in intermediate outputs caused by the adapter, a learnable parameter is used. This learnable parameter also controls the amount of personalized injection at different layers of the network, thereby achieving finer-grained control.

[0034] This invention designs a generative regularizer where, without a conditional LQ image, the face image restoration model is forced to generate identities using only reference images. This approach encourages the face image restoration model to learn robust neural representations of identity, much like generating personalizations. Furthermore, for the base face restoration model, this invention fine-tunes a general restoration model on a face dataset to improve its restoration capabilities.

[0035] During training, instead of resizing and aligning all image samples to a specific size, this invention uses randomly cropped face images. This has several advantages: the face image inpainting model can access higher resolution patches; the face image inpainting model is more robust to different poses; and the face image inpainting model can achieve super-resolution by using tiling.

[0036] It should be noted that, in combination Figure 2 As shown: Figure 2 The left side shows that this face image inpainting model restores the image through a diffusion process conditioned on low-quality (LQ) image samples and high-quality face reference image samples. The base model block is shown in blue, and the personalization module is shown in green. Stable Diffusion is used to extract multi-scale reference features from the reference image samples. During training, reference image samples are randomly sampled from a set of reference image samples in each training iteration. Figure 2 The right side represents a UNet block containing a base model block and personalized modules. The base model block contains a normal stable diffusion block with SFT (Spatial Feature Transform) from StableSR. Following the base model block are intermediate features... Enter a trainable, personalized module containing text embeddings and multi-scale reference features from a reference image. Cross-attention between them. Learnable adapter vectors. It balances the contributions of the base model and personalization.

[0037] In some embodiments of the present invention, the diffusion model includes at least: a basic model block, a personalization module, and a decoder; The face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples, including: Multi-scale reference features of the reference image samples are extracted, structural features of the low-quality image samples are extracted, and initial noise processing is performed on the low-quality image samples to obtain an initial noise image. Acquire real face image samples and add noise to the real face image samples to generate real noise images at different time steps; The real noise image and the conditional vector are input into the base model block, and the intermediate features of the image are output; wherein, the conditional vector is generated based on the structural features and is used to provide multimodal conditional signals for the diffusion model; The image intermediate features and the multi-scale reference features are input into the personalization module for personalized modulation processing, and personalized features are output. Based on the real noise image and the personalized features, the predicted noise is determined. The initial noisy image and the predicted noise are input into the decoder for decoding processing, and a face restoration image sample is output. Based on the total loss function of the face image restoration model, iterative training is performed by inputting the reference image sample and the low-quality image sample into the diffusion model and outputting the face restoration image sample model until a preset termination condition is met, thus obtaining the trained face image restoration model; wherein, meeting the preset termination condition is characterized by the current iteration number reaching a preset number and / or the value of the total loss function being minimized.

[0038] It is understandable that extracting multi-scale reference features from reference image samples includes: Multiple reference image samples are acquired and input into the encoder of a pre-trained diffusion model for encoding processing to obtain the multi-scale reference features.

[0039] Specifically, this can involve using a pre-trained (diffusion model) Stable Diffusion encoder to extract multi-scale reference features (such as detailed texture, local structure, and global layout). : In the formula, Represented as a multi-scale reference feature, Represented as a reference image sample, This is represented as the Stable Diffusion encoder function, where N is the number of layers in the Unet.

[0040] Furthermore, feature aggregation processing can be performed on multi-scale reference features: that is, compact identity embeddings can be generated through cross-layer zero-attention pooling. Used for subsequent condition control.

[0041] Multiscale features Perform cross-layer attention computation to generate a unified identity embedding. .

[0042] formula: , where AttnPool is the attention pooling function.

[0043] Unified identity embedding Function: Compress multi-scale features while retaining key identity information for subsequent condition control.

[0044] Furthermore, given the necessity of prioritizing personalization, the restoration of low-quality images is naturally an ill-posed problem. Let's assume a degradation function... It accepts a facial image. and a degradation value Degradation value The higher the value, the greater the degradation of the output image. As the value approaches infinity, the resulting image will resemble pure noise, and it will no longer be possible to faithfully reconstruct the image. ,in It is a function that returns the identity of a person's face image. It is a recovery model. There exists a... The value exceeds which faithful restoration is no longer possible. However, through additional personalization... It can faithfully restore: Due to identity embedding Not affected by degradation value Regardless of any value, this invention uses a diffusion model to learn an individual's prior identity embedding from a set of reference images. .

[0045] This involves inputting a set of high-quality reference images of the same person (e.g., 3-5 clear faces from different angles and lighting), covering the person's key identifying features (facial features, skin tone, hairstyle, etc.). A pre-trained diffusion model (such as StableDiffusion) encoder is then used to process the reference images... Mapping to the latent space to generate latent codes High-level semantic features are extracted from the latent encoding of each reference image. These features are then aggregated using an attention mechanism or mean pooling to obtain a comprehensive identity embedding. .

[0046] Furthermore, in the process of restoring degraded images, identity embedding As a conditional input diffusion model, traditional methods may fail to recover identity features when the input image is severely degraded (e.g., extremely blurred). This ensures that the output face matches the identity in the reference image. High-frequency details in the reference image (such as wrinkles and moles) are captured through... This information is passed to the restoration result to avoid generating false features.

[0047] In some embodiments, the specific steps for extracting the structural features of the low-quality image samples and performing initial noise processing on the low-quality image samples to obtain an initial noisy image are as follows: Low-quality image samples (LQ images) are randomly fused with noise, i.e., based on degraded low-quality image samples... + Gaussian noise During the diffusion process, an initial noise image x(t) is generated: In the formula, This represents the initial noisy image. This indicates low-quality image samples. Represented as Gaussian noise, This represents a weighting coefficient associated with time step t, used to control low-quality image samples. and Gaussian noise In the initial noise image The mixing ratio in.

[0048] Furthermore, for low-quality image samples Structural features (such as facial contours and key point locations) of low-quality image samples are extracted by edge detection or semantic segmentation networks to generate structural feature vector S.

[0049] In the SFT (Spatial Feature Transformation) module: within the basic model block, structural features are transformed. Convert to affine parameters Modulation characteristics: In the formula, This represents the spatial transformation characteristic, where α represents displacement and β represents scaling.

[0050] It should be further explained that real face image samples are obtained, and noise is added to the real face image samples to generate real noise images at different time steps. Real face image samples Adding noise to generate realistic noise images (Corresponding time step) 1): In the formula, Represented as a real noise image. Represented as real human face image samples, It is represented as a weight coefficient related to time step t, used to control the real face image samples. and Gaussian noise In real noise images The mixing ratio in.

[0051] Furthermore, the real noise image and the condition vector are input into the base model block, and the intermediate features of the image are output; wherein, the condition vector is generated based on the structural features and is used to provide multimodal conditional signals for the diffusion model.

[0052] The specific execution steps can be as follows: the condition vector is generated based on the structural features, including: Cross-layer attention computation is performed on the multi-scale reference features to generate a unified identity embedding; The unified identity embedding, the structural features, and the text embedding prompts are concatenated to generate the conditional vector.

[0053] Specifically, for multi-scale features Perform cross-layer attention computation to generate a unified identity embedding. The formula: in, This represents a unified identity embedding, and AttnPool is the attention pooling function.

[0054] Embed the unified identity of the reference image samples Text Embedded Tips Low-quality image coding features corresponding to the structural features S of low-quality image samples Concatenate them into a conditional vector c.

[0055] The function of the condition vector c is to provide multimodal conditional signals (identity, text, noise) for the diffusion model.

[0056] Furthermore, the real noise image The conditional vector c is input to the basic model block, and the intermediate features of the image are output. .

[0057] Furthermore, the intermediate features of the image and the multi-scale reference features are input into the personalization module for personalized modulation processing, and personalized features are output. Based on the real noise image and the personalized features, the predicted noise is determined.

[0058] It should be noted that the intermediate features of the image and the multi-scale reference features are input into the personalization module for personalized modulation processing, and the personalized features are output, including: Get embedded text hints; Calculate the text attention of the text embedding cue and the intermediate features of the image, and calculate the image attention of the multi-scale reference features and the intermediate features of the image; Controllable modulation features are obtained by performing point-by-point multiplication of the learnable adapter vector, the text attention, the image attention, and the intermediate features of the image. The personalized features are obtained by adding the controllable modulation features and the intermediate features of the image point by point.

[0059] It should be noted that the externally input text description (such as "smiling man") is converted into a high-dimensional vector through a pre-trained text encoder (such as CLIP's TextEncoder). This is called a text embedding cue. This vector captures the semantic information of the text and is used for subsequent attention calculations.

[0060] Text encoders are typically multi-layer Transformers that align the output dimensions with image features (e.g., 768 dimensions) and match the number of channels of intermediate features through projection layers.

[0061] Furthermore, embed text prompts Image intermediate features The formula for calculating text attention by performing cross-attention is as follows: in, Represented as text attention, , , ,in , , Represented as a linear projection matrix, This is represented as a text-embedded hint. This is represented as an intermediate feature of the image.

[0062] Text attention is used to align text semantics with image features, highlighting text-related regions (such as the corners of the mouth feature corresponding to "smile"). Multi-scale reference features (e.g., feature maps of different convolutional layers) and intermediate features of the image The formula for calculating self-attention is as follows: in, Represented as image attention, , , ,in , , Represented as a linear projection matrix, Represented as a multi-scale reference feature, This is represented as an intermediate feature of the image.

[0063] Image attention is used to fuse multi-scale reference features (such as texture and structure) to enhance detail recovery capabilities.

[0064] The formula for the controllable modulation feature is obtained by performing point-by-point multiplication of the learnable adapter vector, the text attention, the image attention, and the intermediate features of the image: In the formula, This is represented as a controllable modulation feature. Represented as intermediate features of the image, These are learnable adapter vectors (channel weights) used to control the intensity of personalization. Represented as text attention, This represents image attention, and ⊙ represents element-wise multiplication.

[0065] By introducing learnable adapter vectors, this can be achieved with a small number of parameters (such as 1x1 convolution), avoiding the introduction of complex computations.

[0066] The personalized feature is obtained by adding the controllable modulation feature and the intermediate image feature point by point, as shown in the following formula: In the formula, This is represented as a personalized feature, specifically the output feature of the i-th layer after cross-attention enhancement and adaptive weighted fusion. This is represented as a controllable modulation feature. This is represented as an intermediate feature of the image.

[0067] This step allows for personalized restoration by overlaying modulation information of text and reference features while preserving the original image structure.

[0068] Based on the real noise image and the personalized features Determine the prediction noise at prediction time step t1 .

[0069] In some embodiments, the spatial transformation features are computed. With multi-scale reference features Cross attention: Adaptive fusion: through learnable weights Mixed characteristics: In the formula, This is represented as the output feature of the i-th layer after cross-attention enhancement and adaptive weighted fusion, which contains information from both the original spatial transformation features and the complementary information from multi-scale reference features. Represented as a learnable adapter vector, Represented as spatial transformation characteristics, It is represented as a multi-scale reference feature.

[0070] Noise prediction: Unet decoder integrates all layers Output prediction noise .

[0071] Furthermore, the initial noisy image and the predicted noise are input into the decoder for decoding processing, and a face restoration image sample is output.

[0072] UNet employs a multi-layered block-based denoising approach, with each layer undergoing processing via a base model block and a personalized module. The final features are then processed by a decoder. Convert to face restoration image samples .

[0073] Furthermore, based on the total loss function used to train the face image restoration model, the model is iteratively trained by inputting the reference image sample and the low-quality image sample into the diffusion model and outputting the face restoration image sample, until a preset termination condition is met, thus obtaining the trained face image restoration model; wherein, meeting the preset termination condition is characterized by the current iteration number reaching a preset number and / or the value of the total loss function being minimized.

[0074] In some embodiments of the present invention, the formula for the total loss function for training the face image restoration model is: In the formula, Represented as the total loss function, This represents the improved diffusion model loss after adding low-quality image samples and reference image samples. This indicates the weights that control the generated items. Represented as regularization loss, This is represented as a cross-attention graph used to standardize learnable text embedding cue tags.

[0075] During the fine-tuning of the personalization module, the model synthesizes low-quality image samples. and from the reference image sample set Sampled reference image As input. Use added low-quality image samples. Improved diffusion model loss for reference image samples: In the formula, This represents the improved diffusion model loss after adding low-quality image samples and reference image samples. It is a diffusion model. It is the latent code of time t. It is a condition vector. It is sampling noise from an isotropic Gaussian distribution.

[0076] In personalized processing, this invention utilizes learnable text embedding and reference image features. and layers Intermediate recovery image features Cross-attention between text and images was employed, such as Figure 2 As shown on the right. Reference image features. They are obtained from frozen stable diffusions, and in practice, they are fed along with LQ image samples through a portion of the base model, known as personalized blocks.

[0077] Controlled Adaptation: Simply adding personalization blocks can lead to output distortion. To avoid excessive changes to the output by personalization blocks, learnable vectors can be used. This initializes the adapter's output. To further control the personalization effect, this invention introduces a separate personalization block for each different resolution. Mathematically speaking, the output of each layer can be represented as: For the first The input feature map of the layer, The feature map corresponding to the reference image in the same network layer i is called the multi-scale reference feature. It is a learnable adapter vector (channel weight vector). The output feature map of the i-th layer is a fusion of degraded image features and personalized priors.

[0078] In some embodiments, generation regularization: compared to personalized generation models in the prior art, the personalized restoration model of the present invention has an additional signal, namely, low-quality images. It guides the restoration of the overall structure of the output, and depending on the severity of the degradation, it may contain some information from the identity. During training, additional input can make the task of reconstructing personalized images from the output easier. However, leaking identity information in the input may prevent the model from fully learning the representation of the identity during training, resulting in poor performance in situations where it is difficult to see the identity.

[0079] To alleviate the aforementioned problems, this invention proposes a generative regularizer to encourage the model to learn more robust identity representations. A regularization loss is added to the original training loss, where zero input is given. Adjusting the LQ image: In the formula, Indicates the regularization loss. It is a diffusion model. It is the latent code of time t. It is a condition vector. It is sampling noise from an isotropic Gaussian distribution.

[0080] This forces the model to generate identity illusions entirely without the aid of conditional images, thereby encouraging more robust representations of identity.

[0081] In some embodiments of the present invention, the present invention uses the face dataset FFHQ to train the model according to the following steps, the dataset being initialized from the pre-trained StableSR.

[0082] Prior knowledge: This invention uses a recovery model, StableSR, pre-trained on general natural images to begin training on face images. Since the model is not trained from scratch on a new task, training time is reduced, and the model is more robust.

[0083] Unaligned methods: Cropping and alignment are commonly used in face processing to standardize the input. However, fine-grained cropping and alignment using face landmarks is prone to errors when face detection models fail. This is especially true in real-world images. To avoid this approach, this invention follows the StableSR training strategy, combining random cropping and resizing for training. Random cropping makes the model more robust while providing higher-resolution input because details are not lost during resizing operations.

[0084] This invention presents a customized facial image restoration model that combines facial restoration and super-resolution personalization techniques. This enables the model to reconstruct images of specific individuals while generating realistic images and maintaining fidelity to the person's identity. The restoration model is personalized using images related to the identity, thus preserving fine-grained details and providing a tailored restoration for each individual.

[0085] Personalization using independent trainable blocks allows for full utilization of the rich priors of the base restoration model. To avoid the face image restoration model relying on conditionally reproducing partial identities in low-quality images, a generative regularizer is employed. Through a learnable parameter, the model learns to strike a balance between detail and personalization generated based on the input image. The training pipeline for the face restoration model is improved to achieve an alignment-free approach.

[0086] The method proposed in this invention can perform customized facial feature restoration for specific individuals, preserving their unique identity information. It also combines deep learning and image processing techniques to effectively restore low-quality or damaged facial images, improve image resolution, and generate high-resolution facial images. Furthermore, it utilizes contextual information from the image for restoration, ensuring the overall consistency and naturalness of the restored results.

[0087] This invention provides a personalized face image restoration method, apparatus, device, medium, and program product. It acquires a low-quality image and a reference image; inputs the low-quality image and the reference image into a face image restoration model for restoration processing, and outputs a personalized restored face image. The face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples. The restored face image is characterized as a personalized image with the same facial structure as the low-quality image and the same facial identity as the reference image. This invention addresses the shortcomings of existing technologies, such as insufficient restoration accuracy and low generalization ability for restoring low-quality face images with varying degrees of degradation. It achieves a balance between general structural fidelity and identity detail control through a well-trained face image restoration model design, making it suitable for restoration tasks of severely degraded images with high generalization ability.

[0088] The personalized face image restoration device provided by the present invention will be described below. The personalized face image restoration device described below can be referred to in correspondence with the personalized face image restoration method described above.

[0089] like Figure 3 The diagram shown is a structural schematic of a personalized face image restoration device provided by the present invention. The personalized face image restoration device includes the following modules: Acquisition module 310 is used to acquire low-quality images and reference images; The repair module 320 is used to input the low-quality image and the reference image into a face image repair model for repair processing, and output a personalized face repair image; wherein, the face image repair model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples; the face repair image is characterized as a personalized image with the same face structure as the low-quality image and the same face identity as the reference image.

[0090] The personalized face image restoration device provided by the present invention is specifically used in the diffusion model, which includes at least: a basic model block, a personalized module, and a decoder; Multi-scale reference features of the reference image samples are extracted, structural features of the low-quality image samples are extracted, and initial noise processing is performed on the low-quality image samples to obtain an initial noise image. Acquire real face image samples and add noise to the real face image samples to generate real noise images at different time steps; The real noise image and the conditional vector are input into the base model block, and the intermediate features of the image are output; wherein, the conditional vector is generated based on the structural features and is used to provide multimodal conditional signals for the diffusion model; The image intermediate features and the multi-scale reference features are input into the personalization module for personalized modulation processing, and personalized features are output. Based on the real noise image and the personalized features, the predicted noise is determined. The initial noisy image and the predicted noise are input into the decoder for decoding processing, and a face restoration image sample is output. Based on the total loss function of the face image restoration model, iterative training is performed by inputting the reference image sample and the low-quality image sample into the diffusion model and outputting the face restoration image sample model until a preset termination condition is met, thus obtaining the trained face image restoration model; wherein, meeting the preset termination condition is characterized by the current iteration number reaching a preset number and / or the value of the total loss function being minimized.

[0091] The personalized face image restoration device provided by the present invention is also specifically used to obtain embedded text prompts; Calculate the text attention of the text embedding cue and the intermediate features of the image, and calculate the image attention of the multi-scale reference features and the intermediate features of the image; Controllable modulation features are obtained by performing point-by-point multiplication of the learnable adapter vector, the text attention, the image attention, and the intermediate features of the image. The personalized features are obtained by adding the controllable modulation features and the intermediate features of the image point by point.

[0092] The personalized face image restoration device provided by the present invention is further used to perform cross-layer attention calculation on the multi-scale reference features to generate a unified identity embedding; The unified identity embedding, the structural features, and the text embedding prompts are concatenated to generate the conditional vector.

[0093] The personalized face image restoration device provided by this invention further includes a formula for the total loss function used to train the face image restoration model: In the formula, Represented as the total loss function, This represents the improved diffusion model loss after adding low-quality image samples and reference image samples. This indicates the weights that control the generated items. Represented as regularization loss, This is represented as a cross-attention graph used to standardize learnable text embedding cue tags.

[0094] The personalized face image restoration device provided by the present invention is further used to acquire multiple reference image samples; The multiple reference image samples are input into the encoder of the pre-trained diffusion model for encoding processing to obtain the multi-scale reference features.

[0095] This invention provides a personalized face image restoration method, apparatus, device, medium, and program product. It acquires a low-quality image and a reference image; inputs the low-quality image and the reference image into a face image restoration model for restoration processing, and outputs a personalized restored face image. The face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples. The restored face image is characterized as a personalized image with the same facial structure as the low-quality image and the same facial identity as the reference image. This invention addresses the shortcomings of existing technologies, such as insufficient restoration accuracy and low generalization ability for restoring low-quality face images with varying degrees of degradation. It achieves a balance between general structural fidelity and identity detail control through a well-trained face image restoration model design, making it suitable for restoration tasks of severely degraded images with high generalization ability.

[0096] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a personalized face image restoration method. This method includes: acquiring a low-quality image and a reference image; inputting the low-quality image and the reference image into a face image restoration model for restoration processing, and outputting a personalized face restoration image; wherein the face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples; and the face restoration image is characterized as a personalized image having the same facial structure as the low-quality image and the same facial identity as the reference image.

[0097] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the personalized face image restoration method provided by the above methods. The method includes: acquiring a low-quality image and a reference image; inputting the low-quality image and the reference image into a face image restoration model for restoration processing, and outputting a personalized face restoration image; wherein the face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples; and the face restoration image is characterized as a personalized image having the same face structure as the low-quality image and the same face identity as the reference image.

[0099] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a personalized face image restoration method provided by the methods described above. This method includes: acquiring a low-quality image and a reference image; inputting the low-quality image and the reference image into a face image restoration model for restoration processing, and outputting a personalized face restoration image; wherein the face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples; and the face restoration image is characterized as a personalized image having the same facial structure as the low-quality image and the same facial identity as the reference image.

[0100] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A personalized face image restoration method, characterized in that, include: Acquire low-quality images and reference images; The low-quality image and the reference image are input into the face image restoration model for restoration processing, and a personalized face restoration image is output. The face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples. The face restoration image is characterized as a personalized image that has the same facial structure as the low-quality image and the same facial identity as the reference image.

2. The personalized face image restoration method according to claim 1, characterized in that, The diffusion model includes at least: a basic model block, a personalization module, and a decoder; The face image restoration model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples, including: Multi-scale reference features of the reference image samples are extracted, structural features of the low-quality image samples are extracted, and initial noise processing is performed on the low-quality image samples to obtain an initial noise image. Acquire real face image samples and add noise to the real face image samples to generate real noise images at different time steps; The real noise image and the conditional vector are input into the base model block, and the intermediate features of the image are output; wherein, the conditional vector is generated based on the structural features and is used to provide multimodal conditional signals for the diffusion model; The image intermediate features and the multi-scale reference features are input into the personalization module for personalized modulation processing, and personalized features are output. Based on the real noise image and the personalized features, the predicted noise is determined. The initial noisy image and the predicted noise are input into the decoder for decoding processing, and a face restoration image sample is output. Based on the total loss function of the face image restoration model, iterative training is performed by inputting the reference image sample and the low-quality image sample into the diffusion model and outputting the face restoration image sample model until a preset termination condition is met, thus obtaining the trained face image restoration model; wherein, meeting the preset termination condition is characterized by the current iteration number reaching a preset number and / or the value of the total loss function being minimized.

3. The personalized face image restoration method according to claim 2, characterized in that, The step of inputting the intermediate features of the image and the multi-scale reference features into the personalization module for personalized modulation processing and outputting personalized features includes: Get embedded text hints; Calculate the text attention of the text embedding cue and the intermediate features of the image, and calculate the image attention of the multi-scale reference features and the intermediate features of the image; Controllable modulation features are obtained by performing point-by-point multiplication of the learnable adapter vector, the text attention, the image attention, and the intermediate features of the image. The personalized features are obtained by adding the controllable modulation features and the intermediate features of the image point by point.

4. The personalized face image restoration method according to claim 3, characterized in that, The condition vector is generated based on the structural features and includes: Cross-layer attention computation is performed on the multi-scale reference features to generate a unified identity embedding; The unified identity embedding, the structural features, and the text embedding prompts are concatenated to generate the conditional vector.

5. The personalized face image restoration method according to claim 2, characterized in that, The formula for the total loss function used to train the face image restoration model is: In the formula, Represented as the total loss function, This represents the improved diffusion model loss after adding low-quality image samples and reference image samples. This indicates the weight that controls the generated items. Represented as regularization loss, This is represented as a cross-attention graph used to standardize learnable text embedding cue tags.

6. The personalized face image restoration method according to claim 2, characterized in that, The extraction of multi-scale reference features from reference image samples includes: Acquire multiple reference image samples; The multiple reference image samples are input into the encoder of the pre-trained diffusion model for encoding processing to obtain the multi-scale reference features.

7. A personalized facial image restoration device, characterized in that, include: The acquisition module is used to acquire low-quality images and reference images; The repair module is used to input the low-quality image and the reference image into the face image repair model for repair processing, and output a personalized face repair image; wherein, the face image repair model is obtained by personalized training of a pre-trained diffusion model based on low-quality image samples and reference image samples; The face restoration image is characterized as a personalized image that has the same facial structure as the low-quality image and the same facial identity as the reference image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the personalized face image restoration method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the personalized face image restoration method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the personalized face image restoration method as described in any one of claims 1 to 6.

Images