Facial attribute-driven human-pet identity-preserving image generation model

Abstract

This invention belongs to the field of computer technology, specifically a facial attribute-driven image generation model for preserving the identity of a person or pet. The model comprises three modules: an image context content generation module; a facial attribute feature embedding module; and a facial attribute-driven identity feature preservation local redrawing module. This invention aims to generate high-fidelity, high-quality portraits of a given person or pet using an image and descriptive text. The invention generates an image consistent with the user's input text using the image context generation module; it then uses the user-provided image to fine-tune the embedding and redrawing modules at different scales through multi-task fine-tuning; finally, the redrawing module performs local redrawing on the generated image, redrawing the main object in the image as the target person or pet. Extensive experiments demonstrate that this invention reduces training time and improves identity fidelity.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to an identity-preserving image generation system. Background Technology

[0002] In recent years, with the continuous development of deep learning technology, text-to-image generation models (hereinafter referred to as text-to-image models) have received widespread attention and research, resulting in important works such as Imagen[1] and Stable Diffusion[2]. Thanks to the large model scale, text-to-image models such as SDXL[3] and DALL-E 3[4] can generate high-quality images. On this basis, subject-driven image generation has become a new focus. This task takes a specific subject (such as a person or object) as the center and generates a customized image of the specific subject based on text prompts. Identity-preserving image generation is a special case of subject-driven image generation tasks. This task requires generating customized images for a specific person, maintaining the identity features of the given person in the generated image, and keeping their visual features, especially facial features, unchanged. Identity-preserving image generation can be applied in fields such as generating customized portraits, virtual try-on, and animation production, and has broad application prospects and research significance.

[0003] Both subject-driven image generation models and identity-preserving image generation models can generate customized images for specific individuals. Existing methods can be divided into two categories: (1) Fine-tuning using images of a given individual, by changing model weights or finding embeddings of special terms to enable the model to learn the identity features of the given individual. Representative methods include DreamBooth[5] and Textual Inversion[6]. (2) Introducing new modules to encode the visual features of a given individual and using these encoded features to guide the diffusion model to generate customized images of the given individual. This type of method is trained on large-scale datasets and does not require fine-tuning of the model for a specific individual during inference, thus possessing zero-sample instant generation capability. Representative works of this type of method include IP-Adapter[7], BLIP-Diffusion[8], and Photo Maker[9]. Existing methods have made some progress in identity-preserving image generation tasks, but still have limitations. First, both fine-tuning the model and introducing new modules to adjust its structure impair the image generation model's capabilities to some extent, resulting in poor quality generation of image context content other than the person. Second, the identity fidelity of the generated images is not high enough, and the facial details of the people in the generated images still differ significantly from the reference images. Furthermore, although customized image generation for a given pet has broad applications, there is currently a significant lack of work and research on identity-preserving image generation for pets. Summary of the Invention

[0004] The purpose of this invention is to provide a facial attribute-driven human-pet identity preservation image generation model to improve the identity fidelity and consistency of facial detail features of a given person or pet in the generated image.

[0005] The facial attribute-driven human-pet identity-preserving image generation model provided by this invention unifies the generation of identity-preserving images of people and pets into a single model. This model decouples the generation of image context content from the generation process of person or pet identity, using an image context content generation module to generate image regions unrelated to identity, thereby improving the overall quality of the generated images. In addition, it constructs a multi-task fine-tuning facial attribute feature embedding module and a facial attribute-driven identity-preserving local redrawing module at different scales, enabling these modules to learn fine-grained features of the facial attributes of a given person or pet, thereby improving the identity fidelity of a given person or pet and the consistency of facial detail features in the generated images. Specifically, this invention uses an image of a given person or pet as a reference and a given descriptive text as a condition to generate a portrait image of a person or pet with high identity fidelity and high quality; it specifically includes the following three modules: (1) an image context content generation module; (2) a facial attribute feature embedding module (hereinafter referred to as the embedding module); and (3) a facial attribute-driven identity feature preservation local redrawing module (hereinafter referred to as the redrawing module); wherein: the image context content generation module generates a context content image that conforms to the context prompt provided by the user; then, it uses the image of the specified person or pet provided by the user to fine-tune the embedding module and the redrawing module to learn the visual features of the given person or pet; finally, according to the facial prompt input by the user, the redrawing module performs local redrawing of the facial area of ​​the person or pet in the context content image to obtain a portrait image of the specified person or pet.

[0006] In this invention, both the image context content generation module and the redrawing module are based on the diffusion model principle and are used to implement a text-to-image generation model. The principle of the diffusion model is to add Gaussian noise to the input image x0 according to the diffusion rule at a series of time steps t1, t2, ..., T until the image becomes pure noise; then, starting from this pure noise, denoising is gradually performed to restore the original image x0.

[0007] The text-to-image generation model places the noise addition and denoising processes within the latent space and trains a denoising model to achieve progressive denoising. During training, the training image x0 is first encoded into a latent space latent variable z0, and then noise is added to it according to the diffusion rule and time step t to obtain the latent variable z. t Then the hidden variable z t With the corresponding time step t as input to the denoising model, and guided by the image-annotated text data, the latent variable z is predicted. tThe text contains noise. After the denoising process is completed, the latent variables are decoded back into the RGB space to generate the final image. With a trained denoising model, the text-to-image generation model can generate an image by progressively denoising arbitrary Gaussian noise under the guidance of given text.

[0008] In this invention, the image context content generation module is a text-to-image generation model based on the diffusion model principle, specifically including a text segmenter, a text encoder, a variational autoencoder, and a U-Net model for predicting noise. The variational autoencoder comprises an encoder ε and a decoder. These are used to transform an image into latent variables in the latent space and, conversely, to transform latent variables in the latent space into an image. The U-Net model for predicting noise can be further divided into a feature extraction network and a feature fusion network. The former is responsible for extracting features from noisy images, while the latter predicts the noise in an image based on those features.

[0009] In this invention, the redrawing module is a text-to-image local redrawing model based on the diffusion model principle. Its U-Net model has a similar structure to the U-Net model in the image context content generation module, but it is specifically used to predict image noise in local redrawing tasks. Apart from the U-Net model, the other components of the redrawing module are the same as those of the image context content generation model.

[0010] In this invention, the embedding module contains five learnable tensors, which serve as embedding representations of the overall facial region and four facial attribute regions: left eye, right eye, nose, and mouth, respectively, for learning the features of the overall face and each facial attribute region of a given person or pet.

[0011] In this invention, the image context content generation module is responsible for generating the context content of areas in the portrait image that are unrelated to the person or pet's identity. The specific steps are as follows:

[0012] (1) Receive the image context prompt T input by the user. c ;

[0013] (2) Utilizing a frozen pre-trained text-to-image generation model Using the contextual cue word T received in step (1) c Given the condition, generate an image that matches the description of the prompt word as the context content image I.

[0014] In this invention, the embedding module and redrawing module are fine-tuned and trained by automatically constructing multi-task datasets S at different scales. The multi-tasks at different scales include reconstructing the overall facial region of a person or pet and local regions centered on the facial features, specifically including reconstructing the overall face, the left eye region, the right eye region, the nose region, and the mouth region. The specific steps for dataset construction and fine-tuning training are as follows:

[0015] (1) Receive a set of images of a given person or pet from the user, containing 3 to 5 different images of the same subject, and enhance them. The enhanced image set is represented as:

[0016] S = {s1, ..., s} n};

[0017] (2) Using the enhanced image set from step (1), automatically construct the dataset for fine-tuning training for the above multi-task; the multi-task is denoted as:

[0018]

[0019] For the i-th task K i The target detection model

[10] is used to detect the bounding box of the target subject in the j-th image and the facial region (the entire facial region or one of the four facial attribute regions) corresponding to the task; the bounding box is expanded in a predefined way to obtain the diagonal coordinates. Obtain <image index j, diagonal coordinates> Formatting annotations;

[0020] (3) Use the multi-task dataset constructed in step (2) to fine-tune the training of the embedding module and the redrawing module; during the fine-tuning process, set the corresponding learning rate according to the type of the main object (the breed of person or pet) and set the proportion of each task in the total training steps.

[0021] In this invention, the facial attribute feature embedding module constructs embedding representations for the overall face and each facial attribute separately, in order to learn the overall facial features and local detailed features of a given person or pet. This invention optimizes these embeddings through multi-task operations at different scales, with the specific steps as follows:

[0022] (1) Select the learning rate for optimizing the embedding representation based on the category of the main object (person or pet);

[0023] (2) Construct and initialize five learnable tensors as embedding representations of the overall facial region and four facial attribute regions: left eye, right eye, nose, and mouth. This is then integrated with multi-tasking... Correspondingly, it is denoted as

[0024] (3) Using a text encoder Extracting fixed facial cue words T for different tasks f Text embedding;

[0025] (4) For the i-th task K i Using embedding to represent V i Replace the facial cue word T in step (3) f Text embedding of facial region words Corresponding embedding;

[0026] (5) Input the replaced facial prompt words from step (4) into the redrawing module, and use it as a condition to perform end-to-end fine-tuning training on the embedding module and the redrawing module. The loss function is as follows:

[0027]

[0028] Where t is the time step in the diffusion model, and z t For the latent variable at this time step, ∈ θ For the U-Net in the redraw module, ∈ represents random Gaussian noise.

[0029] In this invention, the redrawing module reconstructs or generates a masking region by locally redrawing the face or facial attributes of a given character or pet. It receives different inputs and obtains corresponding outputs during the fine-tuning and inference stages, as detailed below:

[0030] During the fine-tuning phase, the parameters of the variational autoencoder (denoted as ε and the decoder as D) in the redrawing module are fixed, and the U-Net model ∈ in the redrawing module is subjected to multi-task testing at different scales. θ To make fine adjustments, the specific steps are as follows:

[0031] (1) Select the corresponding learning rate and the proportion of each task in the total training steps according to the category of the main object (human or pet);

[0032] (2) For task K i Read the image index j and diagonal coordinates from the corresponding training set. coordinates Enhancement is performed to obtain a mask m, and the masked image s′ j ;

[0033] (3) Use the encoder ε of the variational autoencoder of the redraw module to redraw the image s obtained in step (2). j With the masked image s′ j Encoded as latent variables z0 and ε(s′) j Meanwhile, the mask m is processed to obtain m * ;

[0034] (4) Randomly select a time step t within the range [1,T], and add noise to the latent variable z0 obtained in step (3) according to the diffusion rules to obtain the latent variable z. t The details are as follows:

[0035] z t =α t z0+σ t ∈,

[0036] Where, α t With σ t is the hyperparameter used for adding noise in the diffusion model, and ∈ represents random Gaussian noise;

[0037] (5) The latent variable z obtained in step (4) t The ε(s) obtained in step (3) j ′ ) and m * The links are used as input to the redrawing module U-Net; then the embeddings of the replaced facial prompts in the embedding module are used as text conditions and input to the cross-attention layer of the U-Net model; based on the variables input to the U-Net model, text conditions, and time step t, prediction step (4) is performed to the latent variable z. t The added noise is defined as follows, and the loss function is the distance between the predicted noise value and the true noise value:

[0038]

[0039] Fine-tuning is performed simultaneously on the embedding and redrawing modules for end-to-end training. The fine-tuned U-Net model is used to progressively denoise a given noisy masked image, generating the overall face or facial attributes of the subject in the masked region while keeping the non-masked region unchanged, thus achieving local redrawing.

[0040] During the inference phase, the input to the redraw module is the image context generation module. Based on the context clue word T c Generated context content image Facial cue word T f and the facial region feature embedding obtained after fine-tuning The output is a given portrait image of a person or pet. The specific steps are as follows:

[0041] (1) Use an object detection model to detect the entire facial region of a person or pet in the contextual content image I, and obtain the diagonal coordinates p of the detection box of the region. f ;

[0042] (2) Obtain the diagonal coordinates p from step (1) fExpand outwards to form a square to obtain the coordinates. And according to The image is cropped to a square centered on the face region. And calculate the facial region in the square image. relative coordinates in

[0043] (3) Take the square image centered on the face region obtained in step (2). Scale to the input size of the redraw module and scale relative coordinates proportionally.

[0044] (4) Based on the scaled square image from step (3) and relative coordinates Create a facial region mask m and the image after masking.

[0045] (5) Use the encoder ε of the variational autoencoder of the redraw module to re-encode the masked image obtained in step (4). Encoded as Simultaneously, the mask m is processed to obtain m. * ;

[0046] (6) Combine random noise and the result obtained in step (5) With m * Link the input redraw module to use the facial prompt word T f Embedded representation of the overall face and individual facial attributes After being linked, it serves as a text conditional input to the U-Net cross-attention layer in the redraw module, generating a square identity-preserving image.

[0047] (7) The image of the square obtained in step (6) retains its identity. Scale to the original size of the input model before redrawing and adjust according to coordinates. Paste it into the context content image I, and blend it through interpolation to finally generate an image that preserves the identity of the given person or pet.

[0048] The innovation of this invention lies in:

[0049] (1) The image generation model proposed in this invention decouples the generation of a given topic identity from the generation of image context content, enabling the model to generate high-quality context while achieving high identity fidelity and reducing fine-tuning training time.

[0050] (2) The present invention designs a facial attribute-driven embedding representation learning method and a local redrawing module. By setting different scales for the overall face and each facial attribute, the model can effectively learn the visual features of different scales of the face of a person or pet, strengthen the consistency of the generated image with the given person in detail, and thus improve the identity fidelity. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the fine-tuning process of the present invention, taking the reconstruction task of the mouth region as an example. In the fine-tuning process, the present invention first uses the object detection model

[10] to construct a multi-task dataset S of different scales from the user-input target person or pet image. Then, the model is fine-tuned and trained with the goal of predicting the noise in the noisy masked image, and the facial attribute feature embedding module and the facial attribute driven identity preservation local redrawing module are fine-tuned. Training is performed. Through fine-tuning, embedded representations of the overall face and individual facial attributes are obtained. And fine-tune the redraw module The model parameters are used to learn the overall and detailed facial features of the target person or pet.

[0052] Figure 2 This is a flowchart illustrating the inference stage of the photo image generation process in this invention. The model first uses the image context content generation module. Based on contextual clue T c Generate contextual content image I. Then, use facial cue word T. f Conditional on the embedding representation of the overall face and individual facial attributes, the redraw module is used. The facial area of ​​a person or pet in the context image is locally redrawn to obtain a customized image of the given person or pet. Detailed Implementation

[0053] The specific implementation of this invention is divided into two stages: fine-tuning training and inference. In the fine-tuning training stage, the embedding module and the redrawing module are fine-tuned using images of a specified person or pet provided by the user, learning the visual features of the specified person or pet. In the inference stage, firstly, the image context content generation module generates a context content image based on user-provided context prompts (Content Prompt). Then, combined with user-provided facial prompts (Facial Prompt), the redrawing module locally redraws the facial areas in the context content image to obtain a portrait image of the specified person or pet.

[0054] (i) In the fine-tuning training phase, multi-task datasets of different scales are constructed using images of specified people or pets provided by the user. The embedding and redrawing modules are then fine-tuned and trained. The specific steps are as follows:

[0055] Step 1. Based on a given set of images of people or pets, construct a multi-task training dataset S at different scales for fine-tuning the embedding and redrawing modules. The multi-tasks at different scales include two tasks: reconstruction of the overall facial region and reconstruction of local regions centered on facial features. The reconstruction task centered on facial features is further subdivided into four types: reconstruction of the left eye region, reconstruction of the right eye region, reconstruction of the nose region, and reconstruction of the mouth region. The specific process for constructing the above dataset is as follows:

[0056] (1) Receive a set of images of a given person or pet input by the user, containing 3 to 5 different images of the same subject;

[0057] (2) Perform symmetrical or rotated data augmentation on these images. The augmented image set is represented as follows:

[0058] S = {s1, ..., s} n};

[0059] (3) Further preprocess the enhanced image set obtained in step (2); use the object detection model to detect the facial region of people or pets in the image; then crop the image into a square with the facial region as the center to meet the input requirements of the redrawing module; at the same time, ensure that the facial region occupies a certain proportion in the cropped image; the preprocessed image set is still denoted as S={s1,…,s n};

[0060] (4) Using the preprocessed images from step (3), automatically construct a dataset for fine-tuning training for the above-mentioned multi-task at different scales; the multi-task is denoted as:

[0061]

[0062] Among them, task K0 corresponds to the reconstruction of the entire facial region, and tasks K1, ..., K4 correspond to the reconstruction of the left eye region, right eye region, nose region, and mouth region, respectively; for the i-th task K i The target detection model is used to detect the bounding boxes of the target subject and the facial region (the entire facial region or a certain facial attribute region) corresponding to the task in the j-th image.

[0063] (5) Based on task K i For each type of facial region, the detection box obtained in step (4) is expanded outward according to a predefined ratio to obtain new diagonal coordinates. Get Task K i The training data is labeled in the format <image number j, diagonal coordinates>.

[0064] (6) Use the multi-task dataset S constructed in step (5) to fine-tune the training of the embedding module and the redrawing module; during the fine-tuning process, set the corresponding learning rate according to the type of the main object (the breed of person or pet) and set the proportion of each task in the total training steps.

[0065] Step 2. Construct embedding representations for the overall face and each facial attribute, and optimize these embedding representations on multi-task datasets at different scales to learn the overall facial features and local detailed features of a given person or pet; the specific process is as follows:

[0066] (1) Select the learning rate for optimizing the embedding representation based on the category of the main object (person or pet);

[0067] (2) Construct five learnable tensors as embedding representations of the overall facial region and four facial attribute regions (left eye, right eye, nose, and mouth), and integrate them with multi-task... Correspondingly, it is denoted as

[0068] (3) Use the text embedding of the English name of the face or facial attributes to represent the embedding in step (2). Perform initialization;

[0069] (4) Using a text encoder Extracting fixed facial cue words T for different tasks f Text embedding; for example, facial cues T for human portraits. f For a human in Use the English word to represent the i-th facial region, such as face, nose, etc.; for pets, use the corresponding species to replace the word human.

[0070] (5) For the i-th task K i Using embedding to represent V i Replace the facial cue word T in step (4) f text embedding Corresponding embedding;

[0071] (6) Input the embedded facial prompts from step (5) into the redrawing module, and use them as conditions to perform end-to-end fine-tuning training on the embedding module and the redrawing module. The loss function is as follows:

[0072]

[0073] Where t is the time step in the diffusion model, and z t For the latent variable at this time step, ∈ θ ∈ represents the U-Net of the redraw module, and ∈ represents random Gaussian noise.

[0074] Step 3. In the fixed redraw module, use a variational autoencoder (where the encoder is denoted as ε and the decoder as ε). Given the parameters, the U-Net model in the fine-tuning redrawing module is trained using a multi-task training dataset S at different scales. θ The specific process is as follows:

[0075] (1) Select the corresponding learning rate and the proportion of each task in the total training steps according to the category of the main object (human or pet);

[0076] (2) For task K i Randomly select image s j And read the labeled <image index j, diagonal coordinates> from the corresponding training set.

[0077] (3) For task K i For the image number j, diagonal coordinates read in step (2) Process; based on diagonal coordinates Calculate the side length of the detection box Then, within the range of side length Randomly select a length within a 30% range, and set the diagonal coordinates... Expanding this length outwards yields the mask m and the masked image s′. j ;

[0078] (4) Use the encoder ε of the variational autoencoder of the redraw module to redraw the image s obtained in step (2). j Compared with the masked image s′ obtained in step (3) j Encoded as latent variables z0 and ε(s′) j Meanwhile, the mask m is processed to obtain m * ;

[0079] (5) Randomly select a time step t within the range [1,T], and add noise to the latent variable z0 obtained in step (4) according to the diffusion rules to obtain the latent variable z. t The details are as follows:

[0080] z t =α t z0+σ t ∈

[0081] Where, α t With σ t is the hyperparameter used for adding noise in the diffusion model, and ∈ represents random Gaussian noise;

[0082] (6) The latent variable z obtained in step (5) tThe ε(s) obtained in step (4) j ′ ) and m * The links are used as input to the redrawing module U-Net; then the embeddings of the replaced facial prompts in the embedding module are used as text conditions and input to the cross-attention layer of U-Net; based on the variables input to U-Net, text conditions, and time step t, U-Net is used to predict the latent variable z in step (5). t Gaussian noise is added, and then the distance between the noise prediction value and the true value is used as the loss function, as follows:

[0083]

[0084] Fine-tuning is performed simultaneously on the embedding and redrawing modules for end-to-end training. The fine-tuned U-Net is used to progressively denoise a given noisy masked image, generating the overall face or facial attributes of the subject in the masked region while keeping the non-masked region unchanged, thus achieving local redrawing of the masked region.

[0085] (II) In the inference phase, a contextual content image is generated based on the context prompt provided by the user. Then, based on the facial prompt provided by the user, the facial region in the contextual content image is locally redrawn using the finely tuned embedding module and redrawing module to obtain an image of the specified person or pet. The steps in the inference phase are as follows:

[0086] Step 4. Use the image context content generation module Based on the user-input image context prompt word T c Generate context content image I, the specific process is as follows:

[0087] (1) Receive the image context prompt T input by the user. c ;

[0088] (2) Using a fixed image context content generation module The image context prompt T received by the user in step (1) c Generate contextual cue words T based on text conditions. c A consistent image serves as the contextual content image I.

[0089] Step 5. Using the finely tuned embedding and redrawing modules, locally redraw the facial region of the person or pet in the context content image I from Step 4. The specific process is as follows:

[0090] (1) Use an object detection model to detect the entire facial region of a person or pet in the contextual content image I, and obtain the diagonal coordinates p of the detection box of the region. f ;

[0091] (2) The diagonal coordinates p obtained in step (1) f Expand outwards to form a square to obtain the coordinates. According to coordinates Cropping image I yields a square image centered on the facial region. And calculate the relative coordinates of the facial region in the square image.

[0092] (3) Take the square image centered on the face region obtained in step (2). Scale to the input size of the redraw module and scale relative coordinates proportionally.

[0093] (4) Based on the scaled image from step (3) and relative coordinates Create a facial region mask m, and compare it with the masked image.

[0094] (5) Use the encoder ε of the variational autoencoder of the redraw module to re-encode the masked image obtained in step (4). Encoding as Simultaneously, the mask m is processed to obtain m. * ;

[0095] (6) Combine random noise and the result obtained in step (5) With m * Link the input redraw module to use the facial prompt word T f Embedded representation of the overall face and individual facial attributes After being linked, it serves as the text conditional input to the U-Net cross-attention layer in the redraw module, outputting a square-shaped identity-preserving image.

[0096] (7) The image of the square obtained in step (6) retains its identity. Scale the image to the size before redrawing the input model and interpolate the image. With square image Perform blending; blending is performed in the area outside the redrawn region (i.e., mask m), using the following formula:

[0097]

[0098] Where p is the coordinate of the pixel on the image, and α(p) is the weight fused at that pixel, calculated based on the pixel's position p; this step is to eliminate artifacts and other artificial traces.

[0099] (8) Paste the square image obtained after fusion in step (7) back into the context content image I to finally obtain an image that preserves the identity of the given person or pet.

[0100] References

[0101] [1]Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems35(2022),36479–36494.

[0102] [2]Robin Rombach,Andreas Blattmann,Dominik Lorenz,Patrick Esser,and Ommer.2022.High-resolution image synthesis with latent diffusion models.In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.10684–10695.

[0103] [3]Dustin Podell,Zion English,Kyle Lacey,Andreas Blattmann,TimDockhorn,Jonas Müller,Joe Penna,and Robin Rombach.2023.Sdxl:Improving latentdiffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952(2023).

[0104] [4]OpenAI.Dall·e 3system card.2023.

[0105] [5]Nataniel Ruiz,Yuanzhen Li,Varun Jampani,Yael Pritch,MichaelRubinstein,and Kfir Aberman.2023.Dreambooth:Fine tuning text-to-imagediffusion models for subject-driven generation.In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.22500–22510.

[0106] [6]Rinon Gal,Yuval Alaluf,Yuval Atzmon,Or Patashnik,Amit H Bermano,Gal Chechik,and Daniel Cohen-Or.2022.An image is worth one word:Personalizingtext-to-image generation using textual inversion.arXiv preprint arXiv:2208.01618(2022).

[0107] [7]Hu Ye,Jun Zhang,Sibo Liu,Xiao Han,and Wei Yang.2023.Ip-adapter:Text compatible image prompt adapter for text-to-image diffusion models.arXivpreprint arXiv:2308.06721(2023).

[0108] [8]Dongxu Li,Junnan Li,and Steven Hoi.2024.Blip-diffusion:Pre-trainedsubject representation for controllable text-to-image generation andediting.Advances in Neural Information Processing Systems 36(2024).

[0109] [9]Zhen Li,Mingdeng Cao,Xintao Wang,Zhongang Qi,Ming-Ming Cheng,andYing Shan.2023.Photomaker:Customizing realistic human photos via stacked idembedding.arXiv preprint arXiv:2312.04461(2023).

[0110]

[10] Shilong Liu,Zhaoyang Zeng,Tianhe Ren,Feng Li,Hao Zhang,Jie Yang,Chunyuan Li,Jianwei Yang,Hang Su,Jun Zhu,et al.2023.Grounding dino:Marryingdino with grounded pre-training for open-set object detection.arXiv preprintarXiv:2303.05499(2023)。

Claims

1. A facial attribute-driven image generation model for preserving the identity of human pets, characterized in that, This model unifies the generation of identity-preserving images of people and pets into a single model. It decouples the generation of image contextual content from the generation of the person's or pet's identity, using an image contextual content generation module to generate identity-independent image regions, thereby improving the overall quality of the generated images. Furthermore, it constructs a multi-task fine-tuning facial attribute feature embedding module at different scales and a facial attribute-driven identity-preserving local redrawing module. These modules learn fine-grained features of the facial attributes of a given person or pet, thereby improving the identity fidelity and consistency of facial detail features in the generated images. Specifically, using an image of a given person or pet as a reference and given descriptive text as a condition, it generates high-fidelity and high-quality portrait images of people or pets. The system includes the following three modules: (1) Image context content generation module; (2) Facial attribute feature embedding module, hereinafter referred to as the embedding module; and (3) Facial attribute-driven identity feature preservation local redrawing module, hereinafter referred to as the redrawing module. The image context content generation module generates a context content image that conforms to the context prompt words provided by the user. Then, the embedding module and the redrawing module are fine-tuned using the image of the specified person or pet provided by the user to learn the visual features of the given person or pet. Finally, based on the facial prompt words input by the user, the redrawing module performs local redrawing of the facial area of ​​the person or pet in the context content image to obtain a portrait of the specified person or pet. The image context content generation module and the redrawing module are both based on the diffusion model principle and are used to implement the text-to-image generation model. The principle of the diffusion model is to add Gaussian noise to the input image x0 according to the diffusion rule at a series of time steps t1, t2, ..., T until the image becomes pure noise. Then, starting from the pure noise, the noise is gradually removed to restore the original image x0. The text-to-image generation model places the noise addition and denoising processes in the latent space and trains a denoising model to achieve progressive denoising. During training, the training image x0 is first encoded into a latent variable z0 in the latent space, and then noise is added to it according to the diffusion rule and time step t to obtain the latent variable z. t Then the hidden variable z t With the corresponding time step t as input to the denoising model, and guided by the image-annotated text data, the latent variable z is predicted. t The text contains noise; after the denoising process is completed, the latent variables are decoded back to the RGB space to generate the final image; with the training denoising model, the text-to-image generation model can generate an image by gradually denoising arbitrary Gaussian noise under the guidance of a given text.

2. The face attribute-driven human-pet identity-preserving image generation model according to claim 1, characterized in that, The image context content generation module specifically includes a text segmenter, a text encoder, a variational autoencoder, and a U-Net model for predicting noise; wherein, the variational autoencoder includes an encoder ε and a decoder. These are used to transform images into latent variables in the latent space and to inversely transform latent variables in the latent space into images, respectively. The U-Net model for predicting noise is divided into a feature extraction network and a feature fusion network. The former is responsible for extracting features from images containing noise, while the latter predicts noise in images based on image features. The redrawing module specifically includes a text segmenter, a text encoder, a variational autoencoder, and a U-Net model for predicting noise. The U-Net model is structurally similar to the U-Net model in the image context content generation module, but it is specifically used to predict image noise in local redrawing tasks. Apart from the U-Net model, the other components of the redrawing module are the same as those of the image context content generation model. The embedding module contains five learnable tensors, which serve as embedding representations of the overall facial region and four facial attribute regions: left eye, right eye, nose, and mouth. These tensors are used to learn the features of the overall face and each facial attribute region of a given person or pet.

3. The face attribute-driven human-pet identity-preserving image generation model according to claim 2, characterized in that, The image context content generation module is responsible for generating the context content of areas in the portrait image that are unrelated to the person or pet's identity. The specific steps are as follows: (1) Receive the image context prompt T input by the user. c ; (2) Utilizing a frozen pre-trained text-to-image generation model Using the contextual cue word T received in step (1) c Given the condition, generate an image that matches the description of the prompt word as the context content image I.

4. The face attribute-driven human-pet identity-preserving image generation model according to claim 3, characterized in that, The embedding and redrawing modules are fine-tuned by automatically constructing multi-task datasets S at different scales. The multi-tasks at different scales include reconstructing the overall facial region of a person or pet and local regions centered on the facial features, specifically including reconstructing the overall face, the left eye region, the right eye region, the nose region, and the mouth region. The specific steps for dataset construction and fine-tuning are as follows: (1) Receive a set of images of a given person or pet from the user, containing 3 to 5 different images of the same subject, and enhance them. The enhanced image set is represented as: S={s1,…,s n }; (2) Using the enhanced image set from step (1), automatically construct a dataset for fine-tuning training for the above multi-task; the multi-task is denoted as: For the i-th task K i The object detection model is used to detect the bounding box of the target subject in the j-th image and the facial region corresponding to the task; the bounding box is then expanded in a predefined manner to obtain the diagonal coordinates. Obtain <image index j, diagonal coordinates> >Formatting annotations; (3) Use the multi-task dataset constructed in step (2) to fine-tune the training of the embedding module and the redrawing module; During the fine-tuning process, a corresponding learning rate is set according to the type of the main object, and the percentage of each task in the total training steps is set.

5. The face attribute-driven human-pet identity-preserving image generation model according to claim 4, characterized in that, The facial attribute feature embedding module constructs embedding representations for the overall face and each facial attribute separately to learn the overall facial features and local detailed features of a given person or pet; these embeddings are optimized through multi-tasks at different scales, with the specific steps as follows: (1) Select the learning rate for optimizing the embedding representation based on the category of the subject object; (2) Construct and initialize five learnable tensors as embedding representations of the overall facial region and four facial attribute regions: left eye, right eye, nose, and mouth. This is then used in conjunction with multi-tasking. Correspondingly, it is denoted as (3) Using a text encoder Extracting fixed facial cue words T for different tasks f Text embedding; (4) For the i-th task K i Using embedding to represent V i Replace the facial cue word T in step (3) f Text embedding of facial region words Corresponding embedding; (5) Input the replaced facial prompt words from step (4) into the redrawing module, and use it as a condition to perform end-to-end fine-tuning training on the embedding module and the redrawing module. The loss function is as follows: Where t is the time step in the diffusion model, and z t For the latent variable at this time step, ∈ θ For the U-Net in the redraw module, ∈ represents random Gaussian noise.

6. The face attribute-driven human-pet identity-preserving image generation model according to claim 5, characterized in that, The redrawing module reconstructs or generates a masking region by partially redrawing the face or facial attributes of a given character or pet; it receives different inputs and obtains corresponding outputs during the fine-tuning and inference stages, as detailed below: During the fine-tuning phase, the parameters of the variational autoencoder in the redrawing module are fixed, and the U-Net model in the redrawing module is subjected to multi-task testing at different scales. θ To make fine adjustments, the specific steps are as follows: (1) Select the corresponding learning rate and the proportion of each task in the total number of training steps according to the category of the main object; (2) For task K i Read the image number j and diagonal coordinates from the corresponding training set. >, for coordinates Enhancement is performed to obtain a mask m, and the masked image s′ j ; (3) Use the encoder ε of the variational autoencoder of the redraw module to redraw the image s obtained in step (2). j With the masked image s′ j Encoded as latent variables z0 and ε(s′) j Meanwhile, the mask m is processed to obtain m * ; (4) Randomly select a time step t within the range [1, T], and add noise to the latent variable z0 obtained in step (3) according to the diffusion rules to obtain the latent variable z. t The details are as follows: z t =a t z0+σ t ∈, Where, α t With σ t is the hyperparameter used for adding noise in the diffusion model, and ∈ represents random Gaussian noise; (5) The latent variable z obtained in step (4) t The ε(s′) obtained in step (3) j ) and m * The links are used as input to the redrawing module U-Net; then the embeddings of the replaced facial prompts in the embedding module are used as text conditions and input to the cross-attention layer of the U-Net model; based on the variables input to the U-Net model, text conditions, and time step t, prediction step (4) is performed to the latent variable z. t The added noise is defined as follows, and the loss function is the distance between the predicted noise value and the true noise value: Fine-tuning is performed simultaneously on the embedding module and the redrawing module for end-to-end training. The fine-tuned U-Net model is used to progressively denoise a given noisy masked image, generating the overall face or facial attributes of the main object in the masked area while keeping the non-masked area unchanged, thus achieving local redrawing. During the inference phase, the input to the redraw module is the image context generation module. Based on the context clue word T c Generated context content image Facial cue word T f and the facial region feature embedding obtained after fine-tuning The output is a given portrait image of a person or pet. The specific steps are as follows: (1) Use an object detection model to detect the entire facial region of a person or pet in the contextual content image I, and obtain the diagonal coordinates p of the detection box of the region. f ; (2) Obtain the diagonal coordinates p from step (1) f Expand outwards to form a square to obtain the coordinates. And according to The image is cropped to a square shape centered on the face area. And calculate the facial region in the square image. relative coordinates in (3) Take the square image centered on the face region obtained in step (2). Scale to the input size of the redraw module and scale relative coordinates proportionally. (4) Based on the scaled square image from step (3) and relative coordinates Create a facial region mask m and the image after masking. (5) Use the encoder ε of the variational autoencoder of the redraw module to redraw the masked image obtained in step (4). Encoded as Simultaneously, the mask m is processed to obtain m. * ; (6) Combine random noise and the result obtained in step (5) With m * Link the input redraw module to use the facial prompt word T f Embedded representation of the overall face and individual facial attributes After being linked, it serves as a text conditional input to the U-Net cross-attention layer in the redraw module, generating a square identity-preserving image. (7) The image of the square obtained in step (6) retains its identity. Scale to the original size of the input model before redrawing and adjust according to coordinates. Paste it into the context content image I, and blend it through interpolation to finally generate an image that preserves the identity of the given person or pet.

Images