A portrait generation system and method with multimodal fine-grained identity preservation
By extracting and aligning local fine-grained multimodal facial features and global facial features, the problem of inconsistency between the facial area and the overall facial identity in the prior art is solved, efficient portrait generation is achieved, and the diversity and consistency of facial features is ensured. .
Patent Information
- Application Number
- CN202410500098.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-24
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-04-24
AI Technical Summary
Existing image generation techniques ignore fine-grained information when generating facial portraits, resulting in identity inconsistency between the facial area and the overall face, and existing methods often sacrifice diversity and variability of facial features when improving identity consistency.
The multimodal facial prompt generation module is used to extract local fine-grained multimodal facial features and global facial features, and the identity retention network module is used to align features to generate a portrait image consistent with the identity of the acquired facial image data to ensure that each face is Identity consistency in the region.
It significantly improves identity consistency and facial realism in portrait generation, and can ensure identity consistency and diversity of customized facial generation with only one reference image, while improving generation efficiency.
Smart Images

Figure CN118298065B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image generation, and in particular to a portrait generation system and method with multi-modal fine-grained identity preservation. Background Art
[0002] Image generation technology is a technology that uses computer algorithms to automatically generate images. This technology has made significant progress and has been widely used in many fields, such as game design, film production, advertising, medical imaging analysis, etc.
[0003] Currently, image generation technology has undergone a significant evolution, which is driven by the emergence and progress of large-scale diffusion-based text-to-image models, such as GLIDE (a text-guided diffusion model), DALL-E 2 (a novel image generation model), Imagen (a text-to-image generation model), Stable Diffusion (SD), eDiff-I (a text-to-image generation model), and RAPHAEL (a text-to-image generation model). This progress has spawned a large number of application methods across different scenarios. As the central focus of these application methods, personalized and customized portrait generation has attracted widespread attention in both academic and industrial fields due to its wide applicability in downstream tasks such as e-commerce advertising, personalized gift customization, and virtual try-on.
[0004] The main challenge of customized face generation is to maintain the consistency of facial images with different attributes based on one or more reference images, leading to two key issues: ensuring accurate identity (ID) consistency and achieving high-fidelity and diverse facial details. Current text-to-image models, despite combining structural and content guidance, face limitations in precisely controlling personalized and customized generation, especially in terms of the fidelity of generated images to reference images.
[0005] In order to improve the accuracy and diversity of personalized portrait generation from reference images, many customized methods have emerged to meet users' needs for high-quality customized images. These personalized methods are classified according to whether fine-tuning occurs during inference, resulting in two different types: test-time fine-tuning and direct inference. Test-time fine-tuning: This category includes methods such as TextInversion, HyperDreambooth, and CustomDiffusion. Users can enable personalized generation by providing a set of target ID images for post-training. However, despite achieving commendable high-fidelity results, the quality of the generated output depends on the quality of manually collected data. In addition, manually collecting customized data for fine-tuning introduces labor-intensive and time-consuming aspects, limiting its practicality.
[0006] The existing methods proposed: Direct reasoning: Another class of models, including IP-Adapter, Fastcomposer, Photomaker, and InstantID, adopt a single-stage reasoning approach. These models enhance global ID consistency by using images as conditional inputs or manipulating image trigger words. However, existing methods ignore fine-grained information such as landmarks and facial features; although InstantID improves ID consistency to a certain extent by introducing landmarks, visual cue landmarks limit the diversity and variability of key facial regions, resulting in rigid generated facial features, which in turn leads to identity inconsistency between facial regions and the overall face. Summary of the invention
[0007] In order to solve the problem that the existing image generation technology ignores fine-grained facial information, resulting in identity inconsistency between the facial region and the overall face, the present invention proposes a portrait generation system and method with multimodal fine-grained identity preservation, which can ensure the ID consistency of each facial region and significantly improve the identity consistency and facial realism of portrait generation.
[0008] To achieve the purpose of the present invention, the present invention adopts the following technical solutions:
[0009] A portrait generation system with multimodal fine-grained identity preservation, the system comprising: a facial image data acquisition module, a multimodal facial cue generation module, and an identity preservation network module;
[0010] The facial image data acquisition module is used to acquire facial image data to form a facial image data set;
[0011] The multimodal facial cue generation module extracts local fine-grained multimodal facial features and global facial features in the facial image data according to the acquired facial image data set;
[0012] The identity-preserving network module performs feature alignment of the extracted local fine-grained multimodal facial features with the global facial features to generate a portrait image consistent with the identity of the acquired facial image data;
[0013] The facial image data includes a facial image and its corresponding text description.
[0014] In the above technical scheme, the local fine-grained multimodal facial features and the global facial features extracted by the multimodal facial cue generation module are combined with the global facial features by using the identity preservation network module to perform feature alignment, thereby generating a portrait image that is consistent with the identity of the acquired facial image data; combining the local fine-grained multimodal facial features with the global facial features can improve the fine-grained customized face generation, and ensure the identity consistency and facial realism of the portrait generation even when there is only one reference image, while improving the efficiency of portrait generation.
[0015] Preferably, the multimodal facial cue generation module comprises a fine-grained multimodal feature extraction submodule and a global facial identity feature extraction submodule;
[0016] The fine-grained multimodal feature extraction submodule is used to extract local fine-grained multimodal facial features in facial image data, and the global facial identity feature extraction submodule is used to extract global facial features in facial image data.
[0017] Preferably, the fine-grained multimodal feature extraction submodule includes a text feature encoder, an image feature encoder, and a facial feature encoder;
[0018] The text feature encoder is used to generate a corresponding fine-grained multimodal text embedding according to the text description corresponding to the facial image;
[0019] The image feature encoder is used to generate a corresponding fine-grained multimodal facial embedding according to the facial image;
[0020] The facial feature encoder is used to generate local fine-grained multimodal facial features according to text embedding and facial embedding.
[0021] Preferably, the fine-grained multimodal feature extraction submodule is provided with a multimodal large language model and a face model;
[0022] The multimodal large language model is used to extract facial feature descriptions of facial images from facial image data, and concatenate the extracted facial feature descriptions with the text descriptions to form new facial feature descriptions, and the text feature encoder generates corresponding fine-grained multimodal text embeddings according to the new facial feature descriptions;
[0023] The face model is used to extract a segmentation mask of a local facial region from facial image data, and the image feature encoder generates a corresponding fine-grained multimodal facial embedding according to the segmentation mask of the local facial region;
[0024] The facial area includes eyes, nose, ears, mouth and hair.
[0025] Preferably, the facial feature encoder uses a self-attention mechanism based on text embedding and facial embedding to align global facial features with local facial features and generate local fine-grained multimodal facial features.
[0026] In the above technical solution, the fine-grained multimodal feature extraction submodule extracts the facial feature description of the facial image through the internal multimodal large language model, and splices it with the text description to form a new facial feature description, and then generates the corresponding fine-grained multimodal text embedding through the text feature encoder, which can effectively alleviate the identity inconsistency problem in facial details; then, the fine-grained multimodal feature extraction submodule extracts the segmentation mask of the local facial area through the internal face model, and generates the corresponding fine-grained multimodal facial embedding through the image feature encoder, which can save richer identity information; finally, the facial feature encoder is used to generate local fine-grained multimodal facial features according to the text embedding and the facial embedding, using the self-attention mechanism. The adopted self-attention mechanism can cultivate the prompting ability of images and texts, and can focus on the identity information of key facial areas, effectively preventing the influence of identity inconsistency in facial details.
[0027] Preferably, the global facial identity feature extraction submodule includes an image feature encoder, a face feature extractor and a projection layer;
[0028] The image feature encoder and the face feature extractor are used to extract visual features of the facial image and transmit the visual features to the projection layer;
[0029] The projection layer is used to generate global facial features of the facial image based on the visual features.
[0030] In the above technical solution, the global facial identity feature extraction submodule first extracts the visual features of the facial image through the image feature encoder and the face feature extractor, thereby enhancing the features of the facial image, and then uses a lightweight projection layer to generate the global facial features of the facial image according to the visual features, thereby effectively improving the reliability of the global facial features.
[0031] Preferably, the identity preservation network module is provided with a deep learning model, which adopts a facial attention localization mechanism to align the facial region features corresponding to the local fine-grained multimodal facial features and the global facial features, and generates a portrait image consistent with the identity of the acquired facial image data.
[0032] Preferably, the deep learning model introduces a facial segmentation mask, learns an attention score for enhancing text cross-attention from local fine-grained multimodal facial features, and locates a cross-attention map according to the attention score;
[0033] Based on the cross-attention map and its corresponding facial segmentation mask, a portrait image consistent with the identity of the acquired facial image data is generated.
[0034] In the above technical scheme, the deep learning model set up can adopt the facial attention localization mechanism to align the facial region features of the local fine-grained multimodal facial features with the global facial features, thereby realizing accurate identity preservation of the facial image and ensuring the consistency of the local identity features, so as to generate a more vivid facial portrait image. The facial attention localization mechanism can prevent the mixing of identity information from different facial regions to ensure that identity consistency is retained in each facial region.
[0035] A portrait generation method with multi-modal fine-grained identity preservation is applied to a portrait generation system with multi-modal fine-grained identity preservation, and the method comprises the following steps:
[0036] Acquiring facial image data using a facial image data acquisition module to form a facial image data set, and transmitting the facial image data set to a multimodal facial prompt generation module;
[0037] According to the acquired facial image dataset, a multimodal facial cue generation module is used to extract local fine-grained multimodal facial features and global facial features in the facial image data;
[0038] The extracted local fine-grained multimodal facial features are aligned with the global facial features using the identity-preserving network module to generate a portrait image that is consistent with the identity of the acquired facial image data;
[0039] The facial image data includes a facial image and its corresponding text description.
[0040] In the above technical scheme, the local fine-grained multimodal facial features and global facial features extracted by the multimodal facial cue generation module are combined with the global facial features by the identity preservation network module for feature alignment, thereby generating a portrait image that is consistent with the identity of the acquired facial image data; combining the local fine-grained multimodal facial features with the global facial features can improve the fine-grained customized facial generation, thereby ensuring the identity consistency and facial realism of the portrait generation even when there is only one reference image, and at the same time, improving the efficiency of portrait generation.
[0041] Furthermore, according to the text description corresponding to the facial image, a text feature encoder is used to generate the corresponding fine-grained multimodal text embedding;
[0042] Utilize image feature encoder to generate corresponding fine-grained multimodal facial embeddings based on facial images;
[0043] Based on text embedding and face embedding, a facial feature encoder is used to generate local fine-grained multimodal facial features.
[0044] Extracting visual features of the facial image using an image feature encoder and a face feature extractor, and generating global facial features of the facial image using a projection layer based on the visual features;
[0045] The deep learning model is used to adopt the facial attention localization mechanism to align the facial region features corresponding to the local fine-grained multimodal facial features and the global facial features, and generate a portrait image that is consistent with the identity of the acquired facial image data.
[0046] Compared with the prior art, the present invention has the following beneficial effects:
[0047] The present invention proposes a portrait generation system and method with multimodal fine-grained identity preservation. Firstly, local fine-grained multimodal facial features and global facial features are extracted by a multimodal facial cue generation module, and then the extracted local fine-grained multimodal facial features are combined with the global facial features by using an identity preservation network module to perform feature alignment, thereby generating a portrait image with identity consistent with the acquired facial image data; combining the local fine-grained multimodal facial features with the global facial features can improve fine-grained customized facial generation, so that identity consistency and facial realism of portrait generation can be guaranteed even when there is only one reference image, and at the same time, the efficiency of portrait generation is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 A schematic diagram of the structure of a portrait generation system with multi-modal fine-grained identity preservation provided in an embodiment of the present application;
[0049] Figure 2 A schematic diagram of the principle framework of a portrait generation system with multimodal fine-grained identity preservation provided in an embodiment of the present application;
[0050] Figure 3 A schematic diagram of extracting local fine-grained multimodal facial features provided by an embodiment of the present application;
[0051] Figure 4 A schematic diagram of the principle framework of the facial feature encoder provided in an embodiment of the present application for generating local fine-grained multimodal facial features;
[0052] Figure 5 A flowchart of a portrait generation method with multimodal fine-grained identity preservation provided in an embodiment of the present application;
[0053] Figure 6A schematic diagram of the qualitative comparison results of contextualized samples between the system and other models in the experiments provided in the embodiments of the present application;
[0054] Figure 7 A schematic diagram of the qualitative comparison results of stylization and action instructions of the system and other models in the experiment provided in the embodiment of the present application;
[0055] Figure 8 A schematic diagram of verifying high fidelity and naturalness of a face in an experiment provided in an embodiment of the present application;
[0056] Fig. 9 A schematic diagram of the comparison results between the experiment provided in the embodiment of the present application and other fine-tuning-based models;
[0057] Fig.10 This is a schematic diagram of the results of generating diversified personalized ID images given an input ID image in the experiment provided in an embodiment of the present application. DETAILED DESCRIPTION
[0058] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.
[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.
[0060] Embodiment 1:
[0061] This embodiment provides a portrait generation system with multi-modal fine-grained identity preservation, see Figure 1 The system comprises: a facial image data acquisition module, a multimodal facial cue generation module and an identity preservation network module;
[0062] The facial image data acquisition module is used to acquire facial image data to form a facial image data set;
[0063] The multimodal facial cue generation module extracts local fine-grained multimodal facial features and global facial features in the facial image data according to the acquired facial image data set;
[0064] The identity-preserving network module performs feature alignment of the extracted local fine-grained multimodal facial features with the global facial features to generate a portrait image consistent with the identity of the acquired facial image data;
[0065] The facial image data includes a facial image and its corresponding text description.
[0066] Exemplarily, the facial image dataset obtained is the FGID dataset for measuring fine-grained face generation. The FGID dataset includes 525,258 images, which is a compilation from different public datasets such as FFHQ, CelebA, and SFHQ, including approximately 524,000 training data. The dataset includes facial features and descriptions of the facial region and the entire face, supplemented by a novel fine-grained identity consistency metric, and establishes a comprehensive evaluation framework for fine-grained face generation performance. The limitations of existing datasets for capturing facial details that preserve different identities are addressed.
[0067] As a preferred embodiment, see Figure 2 and Figure 3 , the multimodal facial cue generation module includes a fine-grained multimodal feature extraction submodule and a global facial identity feature extraction submodule;
[0068] The fine-grained multimodal feature extraction submodule is used to extract local fine-grained multimodal facial features in facial image data, and the global facial identity feature extraction submodule is used to extract global facial features in facial image data.
[0069] As a preferred embodiment, see Figure 2 , the fine-grained multimodal feature extraction submodule includes a text feature encoder, an image feature encoder and a facial feature encoder;
[0070] The text feature encoder is used to generate a corresponding fine-grained multimodal text embedding according to the text description corresponding to the facial image;
[0071] The image feature encoder is used to generate a corresponding fine-grained multimodal facial embedding according to the facial image;
[0072] The facial feature encoder is used to generate local fine-grained multimodal facial features according to text embedding and facial embedding.
[0073] As a preferred embodiment, see Figure 2 and Figure 3 The fine-grained multimodal feature extraction submodule is provided with a multimodal large language model (i.e., LLaVA1.5 model) and a face model (i.e., BiSeNet model);
[0074] The multimodal large language model is used to extract facial feature descriptions of facial images from facial image data, and concatenate the extracted facial feature descriptions with the text descriptions to form new facial feature descriptions, and the text feature encoder generates corresponding fine-grained multimodal text embeddings according to the new facial feature descriptions;
[0075] The face model is used to extract a segmentation mask of a local facial region from facial image data, and the image feature encoder generates a corresponding fine-grained multimodal facial embedding (i.e., facial features embedding) according to the segmentation mask of the local facial region;
[0076] The facial area includes eyes, nose, ears, mouth and hair.
[0077] As a preferred embodiment, see Figure 2 ,The facial feature encoder uses a self-attention mechanism to align global facial features with local facial features based on text embedding and facial embedding, and generates local fine-grained multimodal facial features.
[0078] For example, see Figure 2 and Figure 3 , using the command prompt “Describe the facial features of this person, including face, ears, eyes, nose, and mouth”, the entire facial image is input into the multimodal large language model (MLLM) LlaVA1.5 to obtain a facial feature-level description; subsequently, the words “face, ears, eyes, nose, and mouth” in these feature-level descriptions are replaced by the separator “ <facial>", and connect them with the caption of the entire facial image to form a new facial feature description; finally, the new facial feature description is input into the pre-trained text encoder to learn fine-grained multimodal facial features, and the text feature encoder is used to generate the corresponding fine-grained multimodal text embedding according to the new facial feature description; through the visual and text description containing more accurate ID information, the ID (identity) inconsistency problem in facial details is effectively alleviated.
[0079] In order to enrich the ID (identity) preservation information, it is necessary to study more fine-grained facial features, including eye gaze, ear lobe features, nose shape, etc. The pre-trained face model BiSeNet is used to extract the segmentation mask of the facial area from the entire face image, including eyes, nose, ears, mouth and other areas, other areas including hair, other facial features, etc.; then, the facial area image obtained from the extracted segmentation mask is input into the pre-trained image encoder to learn fine-grained facial embedding, and then generate the corresponding fine-grained multimodal facial embedding, such as Figure 2 shown.
[0080] To cultivate the potential of both image and text cues, a face encoder is designed to seamlessly integrate text embeddings with face embeddings along the dimension of text sequence, e.g. Figure 4 Specifically, given a face embedding (i.e., facial features embedding) and a text embedding, the face encoder initially uses a self-attention mechanism to align global facial features with local facial features to obtain aligned features, denoted as Where N = 5 represents the number of facial feature regions, including eyes, mouth, ears, nose, and other facial regions, and D represents the dimension of text embedding. In the case where the face image lacks the complete N facial features, the missing features are filled with a full zero matrix. Subsequently, we will delimiter " <facial>"The text feature at the position is replaced by Two multi-layer perceptrons (MLPs) are then used to learn text-conditional embeddings to generate local fine-grained multimodal facial features.
[0081] In the system, in order to enhance the controllability of the text, it is necessary to prioritize the title as the main prompt, optimize the parameters of the facial encoder and projection model layers, and keep the parameters of the pre-trained diffusion model in a frozen state. The training data consists of facial image-text pairs, which are connected with a more detailed description of the facial area extracted from LlaVA1.5 to form the final text input; Regarding the training loss function, they are aligned with the loss function used in the original stable diffusion model, expressed as:
[0082]
[0083] Among them C i Indicates the facial ID feature, C f is a fine-grained multimodal facial feature, and the total loss function is During inference, a direct delayed primitive conditioning similar to Fastcomposer is adopted, which allows using a separate text representation initially and then enhancing the text representation after a specific step, effectively balancing identity preservation and editability.
[0084] It can be understood that the fine-grained multimodal feature extraction submodule extracts the facial feature description of the facial image through the internal multimodal large language model, and splices it with the text description to form a new facial feature description, and then generates the corresponding fine-grained multimodal text embedding through the text feature encoder, which can effectively alleviate the identity inconsistency problem in facial details; then, the fine-grained multimodal feature extraction submodule extracts the segmentation mask of the local facial area through the internal face model, and generates the corresponding fine-grained multimodal facial embedding through the image feature encoder, which can save richer identity information; finally, the facial feature encoder is used to generate local fine-grained multimodal facial features according to the text embedding and the facial embedding, using the self-attention mechanism. The adopted self-attention mechanism can cultivate the prompting ability of images and texts, and can focus on the identity information of key facial areas, effectively preventing the influence of identity inconsistency in facial details.
[0085] As a preferred embodiment, see Figure 2 , the global facial identity feature extraction submodule includes an image feature encoder, a face feature extractor and a projection layer;
[0086] The image feature encoder and the face feature extractor are used to extract visual features of the facial image and transmit the visual features to the projection layer;
[0087] The projection layer is used to generate global facial features of the facial image based on the visual features.
[0088] Exemplarily, the complete face image is simultaneously input into the image feature encoder and the face feature extractor for visual feature extraction, and then a lightweight projection layer with parameters initialized by IPA-FaceID-Plus is used to generate a face embedding of the entire image to generate global facial features of the face image.
[0089] It can be understood that the global facial identity feature extraction submodule first extracts the visual features of the facial image through the image feature encoder and the face feature extractor, thereby enhancing the features of the facial image, and then uses a lightweight projection layer to generate the global facial features of the facial image according to the visual features, thereby effectively improving the reliability of the global facial features.
[0090] As a preferred embodiment, see Figure 2 The identity preservation network module is provided with a deep learning model, which adopts a facial attention localization mechanism to align the facial region features corresponding to the local fine-grained multimodal facial features and the global facial features, and generates a portrait image consistent with the identity of the acquired facial image data.
[0091] As a preferred embodiment, the deep learning model introduces a facial segmentation mask, learns an attention score for enhancing text cross-attention from local fine-grained multimodal facial features, and locates a cross-attention map according to the attention score;
[0092] Based on the cross-attention map and its corresponding facial segmentation mask, a portrait image consistent with the identity of the acquired facial image data is generated.
[0093] Exemplarily, a deep learning model (i.e., Unet model) is used to adopt a facial attention localization mechanism, and a facial segmentation mask is introduced to align the facial region features corresponding to the local fine-grained multimodal facial features with the global facial features, and the attention score of the enhanced text cross-attention is learned from the local fine-grained multimodal facial features. The process is:
[0094] Let P∈[0,1] h×w×n Represents a cross-attention map that connects latent pixels to the multimodal conditional embedding at each layer, where P[i, j, k] represents the attention map of the k-th conditional label to the (i, j) latent pixel; the attention map of the facial region label should only focus on the facial feature area, preventing identity mixing between facial features and reverse propagation to the entire facial image; at the same time, the segmentation mask of the reference facial region features is used to locate the cross-attention map.
[0095] Let M = {m 1 , m 2 , m 3 , ..., m N } represents the segmentation mask of the reference portrait, I = {i 1 ,i 2 ,i 3 , ..., i N } as a list of indices indicating which facial features correspond to the visual and textual markers in the multimodal cue, P i =P[:,:,i]∈[0,1] h×w Represents the cross attention map of the i-th facial region label, where I uses a special label" <face>"Generate; Given a cross attention map Making the Cross Attention Map corresponds closely to the facial region identified by the jth multimodal marker, represented by m j To this end, we introduce m j and apply it to Get the corresponding activation area and the segmentation mask m of the j-th facial feature marker j Alignment; at the same time, a balanced L1 loss is used to minimize the distance between the cross attention map and the segmentation mask:
[0096]
[0097] Among them, N represents the number of segmentation masks. The loss formula can ensure that the attention map of each facial feature marker is closely aligned with its corresponding segmentation mask, promoting accurate and local attention in the portrait generation process to generate a portrait image that is consistent with the identity of the acquired facial image data.
[0098] It can be understood that the set deep learning model can adopt the facial attention localization mechanism to align the facial region features of the local fine-grained multimodal facial features with the global facial features, thereby achieving accurate identity preservation of the facial image and ensuring the consistency of local identity features, so as to generate more vivid facial portraits. The facial attention localization mechanism can prevent the mixing of identity information from different facial regions to ensure that identity consistency is retained in each facial region.
[0099] In this embodiment, the local fine-grained multimodal facial features and the global facial features extracted by the multimodal facial cue generation module are combined with the global facial features by using the identity preservation network module to perform feature alignment, thereby generating a portrait image that is consistent with the identity of the acquired facial image data; combining the local fine-grained multimodal facial features with the global facial features can improve fine-grained customized facial generation, ensure identity consistency and facial realism of portrait generation even when there is only one reference image, and at the same time, improve the efficiency of portrait generation.
[0100] Embodiment 2:
[0101] This embodiment provides a portrait generation method with multi-modal fine-grained identity preservation, see Figure 5 , the method is applied to a portrait generation system with multimodal fine-grained identity preservation, the method comprising the following steps:
[0102] S1: using a facial image data acquisition module to acquire facial image data to form a facial image data set, and transmitting the facial image data set to a multimodal facial prompt generation module;
[0103] S2: according to the acquired facial image dataset, a multimodal facial cue generation module is used to extract local fine-grained multimodal facial features and global facial features in the facial image data;
[0104] S3: Using the identity-preserving network module, the extracted local fine-grained multimodal facial features are aligned with the global facial features to generate a portrait image that is consistent with the identity of the acquired facial image data;
[0105] The facial image data includes a facial image and its corresponding text description.
[0106] In some embodiments, based on the text description corresponding to the facial image, a text feature encoder is used to generate a corresponding fine-grained multimodal text embedding;
[0107] Utilize image feature encoder to generate corresponding fine-grained multimodal facial embeddings based on facial images;
[0108] Based on text embedding and face embedding, a facial feature encoder is used to generate local fine-grained multimodal facial features.
[0109] Extracting visual features of the facial image using an image feature encoder and a face feature extractor, and generating global facial features of the facial image using a projection layer based on the visual features;
[0110] The deep learning model is used to adopt the facial attention localization mechanism to align the facial region features corresponding to the local fine-grained multimodal facial features and the global facial features, and generate a portrait image that is consistent with the identity of the acquired facial image data.
[0111] In this embodiment, the local fine-grained multimodal facial features and the global facial features extracted by the multimodal facial cue generation module are combined with the global facial features by the identity preservation network module for feature alignment, thereby generating a portrait image that is consistent with the identity of the acquired facial image data; combining the local fine-grained multimodal facial features with the global facial features can improve fine-grained customized facial generation, ensure identity consistency and facial realism of portrait generation even when there is only one reference image, and at the same time, improve the efficiency of portrait generation.
[0112] Embodiment three:
[0113] This example provides some experimental data during the experiment of the present invention, which are as follows:
[0114] A dataset pipeline outlined in the supplementary material is introduced to create the dataset FGID, which contains comprehensive fine-grained ID information and detailed face descriptions.
[0115] The FGID dataset includes 525,258 images and is a compilation from different public datasets such as FFHQ, CelebA, SFHQ, including approximately 524,000 training data; the dataset contains full face and facial feature information, providing richer text and visual details for model training. Text data: For textual facial description, LLAAVA1.5 (MLLM) is used to use the embedding prompt "Please describe the person in the image, including their gender, age, clothing, facial expression, or any other distinguishing features". Visual data: BiSeNet and InsightFace (face feature extractor) are deployed to capture the entire face ID information and facial feature information, ensuring comprehensive identity details.
[0116] See also Figure 6 , a qualitative evaluation was conducted, and ConsistentID (i.e., a portrait generation system with multimodal fine-grained identity preservation) was compared with Fastcomposer, IP-Adapter, Photomaker, and InstantID. All used one reference image for personalized generation, used default parameters for each method, and restricted reasoning to a single reference image, aligned with the Photomaker method, and quantitatively evaluated using the Mystyle dataset and combined with more than 10 identity datasets for visualization; through comparison, it was demonstrated that ConsistentID exhibits stronger capabilities in high-quality generation, flexible editability, and strong identity fidelity.
[0117] See also Figure 7 , a quantitative evaluation is conducted, and ConsistentID is qualitatively compared with other models on two special tasks, stylization and action instructions. The benchmark indicators evaluate text consistency (CLIP-T), preservation of coarse-grained and fine-grained ID information (CLIP-I, DINO, FaceSIM and FGIS), generation quality (FID) and inference efficiency (speed in seconds). Following Photomaker, a test dataset from Mystyle is used, and facial descriptions are obtained during inference using LlaVA1.5 (MLLM). The quantitative comparison is conducted under a general recontextualization setting, using a set of indicators to benchmark various aspects; the results are shown in Table 1. The analysis in Table 1 shows that ConsistentID consistently outperforms other methods in most evaluation indicators, and exceeds other IP-Adapter-based methods in terms of generation efficiency, demonstrating the fine-grained ID preservation capability of ConsistentID and the efficiency of the lightweight multimodal facial prompt generator. The portraits generated by ConsistentID are more realistic and more consistent in identity.
[0118] Table 1:
[0119]
[0120] See also Figure 8 ,Based on two downstream applications, the aforementioned models are compared,and compared with Photomaker and InstantID,ConsistentID shows strong ability in maintaining ID consistency,,high facial fidelity, and natural expressions, demonstrating the effective,fine-grained ID preservation of ConsistentID’s facial features.
[0121] See also Fig. 9 ,Comparison between ConsistentID and other fine-tuning based models, the results show that the ,ConsistentID proposed in the present invention has better portrait ,generation effect.
[0122] See also Fig.10 ,Given some input ID images, ConsistencyID can generate diverse personalized ID images based on text ,hints using only a single image.
[0123] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.< / face> < / facial> < / facial>
Claims
1. A portrait generation system with multimodal fine-grained identity preservation, characterized in that The system comprises: a facial image data acquisition module, a multimodal facial cue generation module and an identity preservation network module; The facial image data acquisition module is used to acquire facial image data to form a facial image data set; The multimodal facial cue generation module extracts local fine-grained multimodal facial features and global facial features in the facial image data according to the acquired facial image data set; The identity-preserving network module performs feature alignment of the extracted local fine-grained multimodal facial features with the global facial features to generate a portrait image consistent with the identity of the acquired facial image data; Wherein, the facial image data includes a facial image and its corresponding text description; The multimodal facial cue generation module includes a fine-grained multimodal feature extraction submodule and a global facial identity feature extraction submodule; The fine-grained multimodal feature extraction submodule is used to extract local fine-grained multimodal facial features in facial image data, and the global facial identity feature extraction submodule is used to extract global facial features in facial image data; The fine-grained multimodal feature extraction submodule includes a text feature encoder, an image feature encoder, and a facial feature encoder; The text feature encoder is used to generate a corresponding fine-grained multimodal text embedding according to the text description corresponding to the facial image; The image feature encoder is used to generate a corresponding fine-grained multimodal facial embedding according to the facial image; The facial feature encoder is used to generate local fine-grained multimodal facial features based on text embedding and facial embedding; The global facial identity feature extraction submodule includes an image feature encoder, a face feature extractor, and a projection layer; The image feature encoder and the face feature extractor are used to extract visual features of the facial image and transmit the visual features to the projection layer; The projection layer is used to generate global facial features of the facial image based on the visual features.
2. The portrait generation system with multimodal fine-grained identity preservation according to claim 1, characterized in that The fine-grained multimodal feature extraction submodule is provided with a multimodal large language model and a face model; The multimodal large language model is used to extract facial feature descriptions of facial images from facial image data, and concatenate the extracted facial feature descriptions with the text descriptions to form new facial feature descriptions, and the text feature encoder generates corresponding fine-grained multimodal text embeddings according to the new facial feature descriptions; The face model is used to extract a segmentation mask of a local facial region from facial image data, and the image feature encoder generates a corresponding fine-grained multimodal facial embedding according to the segmentation mask of the local facial region; The facial area includes eyes, nose, ears, mouth and hair.
3. The portrait generation system with multimodal fine-grained identity preservation according to claim 2, characterized in that The facial feature encoder uses a self-attention mechanism based on text embedding and facial embedding to align global facial features with local facial features and generate local fine-grained multimodal facial features.
4. The portrait generation system with multimodal fine-grained identity preservation according to claim 1, characterized in that The identity preservation network module is provided with a deep learning model, which adopts a facial attention localization mechanism to align the facial region features corresponding to the local fine-grained multimodal facial features and the global facial features, and generates a portrait image consistent with the identity of the acquired facial image data.
5. The portrait generation system with multimodal fine-grained identity preservation according to claim 4, characterized in that The deep learning model introduces facial segmentation masks, learns attention scores for enhancing text cross-attention from local fine-grained multimodal facial features, and locates cross-attention maps based on the attention scores; Based on the cross-attention map and its corresponding facial segmentation mask, a portrait image consistent with the identity of the acquired facial image data is generated.
6. A method for generating a portrait with multimodal fine-grained identity preservation, the method being used to implement the function of the system as claimed in any one of claims 1 to 5, characterized in that: The method comprises the following steps: Acquiring facial image data using a facial image data acquisition module to form a facial image data set, and transmitting the facial image data set to a multimodal facial prompt generation module; According to the acquired facial image dataset, a multimodal facial cue generation module is used to extract local fine-grained multimodal facial features and global facial features in the facial image data; The extracted local fine-grained multimodal facial features are aligned with the global facial features using the identity-preserving network module to generate a portrait image that is consistent with the identity of the acquired facial image data; The facial image data includes a facial image and its corresponding text description.
7. The method for generating a portrait with multimodal fine-grained identity preservation according to claim 6, characterized in that: Based on the text description corresponding to the facial image, a text feature encoder is used to generate the corresponding fine-grained multimodal text embedding; Utilize image feature encoder to generate corresponding fine-grained multimodal facial embeddings based on facial images; Based on text embedding and face embedding, a facial feature encoder is used to generate local fine-grained multimodal facial features. Extracting visual features of the facial image using an image feature encoder and a face feature extractor, and generating global facial features of the facial image using a projection layer based on the visual features; The deep learning model is used to adopt the facial attention localization mechanism to align the facial region features corresponding to the local fine-grained multimodal facial features and the global facial features, and generate a portrait image that is consistent with the identity of the acquired facial image data.
