Consistency identity picture generation method based on diffusion model

Abstract

The invention belongs to the field of computer vision, and particularly relates to a consistent identity picture generation method based on a diffusion model, which comprises the following steps of: generating optimal paired data by adopting a multi-modal large language model, the paired data comprising text data and corresponding image data; carrying out encoding processing on the text data by adopting a text encoder to obtain a text embedding vector; processing the image data by adopting an image encoder to obtain an image embedding vector; inputting the text embedding vector and the image embedding vector into an MLP module for fusion to obtain a fusion embedding vector; stacking all the fusion embedding vectors, and processing the stacked fusion embedding vectors by adopting a cross attention layer to obtain deep-level features; performing multi-stage adjustment on the deep feature to obtain a consistent identity picture; according to the invention, a coarse-to-fine multi-stage fine adjustment method is adopted, and the visual quality of the generated image is gradually optimized.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a method for generating consistent identity pictures based on a diffusion model. Background Art

[0002] Diffusion models have demonstrated excellent performance in a variety of image generation tasks, but current personalized generation methods still have many shortcomings. Existing techniques include tuning-based and non-tuning personalized generation methods. Tuning-based personalized generation methods, such as Textual Inversion and DreamBooth, usually fine-tune some parameters of the diffusion model to preserve the identity of the reference image, but these methods often require training a new model for each new user, which is inefficient and lacks generalization. In addition, such methods will have a "copy-paste" effect when generating images, making it difficult to generate images that follow complex prompts while maintaining identity. In recent years, non-tuning personalized generation methods have begun to attract attention. Such methods extract the visual embedding of the reference image and inject it into the diffusion process, attempting to achieve personalized image generation without the need for individual tuning for each user. However, existing non-tuning methods also have the problems of insufficient generation diversity and poor ability to cope with complex prompts. Summary of the invention

[0003] To solve the problems existing in the above prior art, the present invention proposes a method for generating a consistent identity image based on a diffusion model, which includes: using a multimodal large language model to generate optimal paired data, wherein the paired data includes text data and corresponding image data; using a text encoder to encode the text data to obtain a text embedding vector; using an image encoder to process the image data to obtain an image embedding vector; inputting the text embedding vector and the image embedding vector into an MLP module to enter fusion to obtain a fused embedding vector; stacking all the fused embedding vectors, and using a cross-attention layer to process the stacked fused embedding vectors to obtain deep-level features; performing multi-stage adjustments on the deep-level features to obtain a consistent identity image.

[0004] Beneficial effects of the present invention:

[0005] The present invention enhances the diversity of generated images through a new synthetic paired data generation mechanism; the present application uses a fully parallel attention architecture combined with three text encoders and a trainable visual encoder to improve the alignment of text and image; the present application adopts a coarse-to-fine multi-stage fine-tuning method to gradually optimize the visual quality of generated images; the present application achieves highly consistent image and video generation in terms of scalability and adaptability, so that the model can meet the personalized needs of multiple users. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A flow chart for processing data for the diffusion model of the present invention;

[0007] Figure 2 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0008] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0009] A consistent identity image generation method based on diffusion model, such as Figure 2 As shown, the method includes: using a multimodal large language model to generate optimal paired data, wherein the paired data includes text data and corresponding image data; using a text encoder to encode the text data to obtain a text embedding vector; using an image encoder to process the image data to obtain an image embedding vector; inputting the text embedding vector and the image embedding vector into an MLP module to fuse them to obtain a fused embedding vector; stacking all the fused embedding vectors, and using a cross-attention layer to process the stacked fused embedding vectors to obtain deep-level features; performing multi-stage adjustments on the deep-level features to obtain a consistent identity picture.

[0010] In this embodiment, using a multimodal large language model to generate optimal paired data includes: obtaining a reference image, inputting the reference image into the large language model, and generating text information; injecting expression and posture information into the text information to obtain multimodal text information; using a text generation model to process the multimodal text information to obtain a synthetic image; preprocessing the synthetic image, and associating the text information with the preprocessed synthetic image to obtain the optimal paired data.

[0011] Specifically, the data generation process includes: improving generation diversity by creating high-quality training data of the same identity but with different expressions, postures, and lighting conditions. The specific process is as follows: Image text description generation: Generate detailed text descriptions of real reference images through a multimodal large language model (such as Llama3); Text description rewriting: Use the description rewriting stage to inject more expression and posture diversity; Text generation image: Use a text generation model (such as Emu) to generate high-quality synthetic images; Image refinement processing: Further enhance the identity consistency of the generated image through reference image preprocessing. The above method can not only generate high-quality paired data, but also effectively avoid the high pairing problem that is difficult to obtain with traditional datasets.

[0012] A text generation model is a model that can generate text based on input contextual information. It captures the laws and features of language by learning from a large amount of text data, thereby achieving the function of automatically generating text. The text generation model can improve its capabilities by learning, fusing, and integrating features from different modalities. For example, in the image description generation task, the model can first extract key features in the image (such as color, shape, texture, etc.), and then combine these features with text features to generate accurate descriptive text. Generate the corresponding synthetic image based on the descriptive text.

[0013] The use of a text generation model to process multimodal text information includes: the text generation model includes an input layer, a hidden layer and an output layer; the multimodal text information is input into the input layer for encoding processing to obtain a feature map; the feature map is input into the hidden layer to extract hidden features; the hidden features are input into the output layer, and after being restored by a decoder, a synthetic image is obtained.

[0014] In this embodiment, based on the input image and text prompt, the content or attributes are generated under the control of the text prompt while retaining the input identity features. Figure 1 As shown in the figure, text and image embedding acquisition: First, text embedding and image embedding are obtained from the text encoder and image encoder. Then, the corresponding category embedding (e.g., male and female) and each image embedding are merged through MLP to extract the fused embedding. The embedding is regularized during this period to enhance its semantic understanding and alignment. Not only is the scale of the embedding vector rescaled in the intermediate optimization stage, but the accuracy and compatibility of text generation are also enhanced by the newly introduced regularization term. The calculation method is as follows: randomly select one from a set of preset prompt templates, such as "a{}in the jungle", and then fill this template with new concept words to form two different prompts. Next, the inputs of these two prompts are fed into a text encoder, and the encoder outputs the corresponding embedding representations. The goal is to minimize the average cosine distance between the two embedding representations to ensure that they are as close as possible in the vector space.

[0015] The text encoder is used to encode text data, including: using CLIP-T as the text encoder, adding position embedding to the text encoder to maintain word order information, and using layer normalization and residual connection to ensure the stability of training; in the last layer of the encoder, the self-attention mechanism is used to represent the features, and the representation results are used as embedding vectors.

[0016] In this embodiment, the CLIP model, namely the Contrastive Language-Image Pre-training model, is a multimodal model based on contrastive learning, which can train a transferable visual model by contrasting text-image pairs. The text is processed using CLIP-T, which mainly includes the following steps: standardizing the input text, including removing blank characters, converting to lowercase, and segmenting words; using CLIPTokenizer to encode the text using BPE (Byte Pair Encoding) to convert the text into a series of token_ids; inputting the encoded token_id into the text encoder of CLIP; the text encoder uses the Transformer model, which can handle long-distance dependencies and effectively capture the semantic information in the text; the text encoder outputs the feature vector of the text, which represents the semantic information of the text.

[0017] During the training process of the CLIP-T model, the model receives a large number of text-image pairs as input; for each text-image pair, the CLIP model calculates the similarity between the text feature vector and the image feature vector; the contrastive loss function is used to optimize the parameters of the CLIP model; the goal of the contrastive loss function is to make the feature vectors of the matching text-image pairs as close as possible; through continuous iterative optimization, the CLIP model can learn the matching relationship between text and image.

[0018] In this embodiment, the image data is processed using an image encoder, including: using CLIP-V as a feature extractor, multi-level features from underlying textures to high-level semantics, and compressing the extracted features into a 1D embedding vector through a global average pooling layer.

[0019] The image encoder (CLIP-V) is one of the core components of the CLIP model, which is responsible for converting the input image data into high-dimensional feature vectors. These feature vectors can capture the key information of the image, such as color, shape, texture, and more advanced semantic features. By comparing these feature vectors with text feature vectors, the CLIP model can learn the association between images and text.

[0020] Using an image encoder to process an image includes: inputting image data into the CLIP model, and the image encoder extracts features from the image through a convolutional neural network (CNN) or other deep learning architectures. The extracted feature vectors are used as representations of the image, which can be used for subsequent tasks such as image classification, retrieval, or cross-modal matching.

[0021] The MLP module fuses the embedded vectors, including a multi-layer perceptron network including multiple hidden layers and a ReLU activation function; the multi-layer perceptron network is used to fuse the features to obtain a fused embedded vector.

[0022] In one embodiment, MLP is composed of multiple neuron layers, including an input layer, one or more hidden layers, and an output layer. The neurons in each layer are fully connected to all neurons in the next layer, and information is transmitted through weights and biases. MLP contains at least three layers of network structure, namely, input layer, hidden layer (can have multiple layers) and output layer.

[0023] Input layer: Receives raw data or features and passes them to the next layer. Each input node represents a feature, and the number of nodes in the input layer is determined by the dimension of the feature.

[0024] Hidden layer: The middle layer connecting the input layer and the output layer. Each hidden layer contains multiple neurons (nodes), each neuron is connected to all nodes in the previous layer, and outputs a weighted and activated value. The number of neurons in the hidden layer and the number of hidden layers can be adjusted as needed, which is also an important hyperparameter in the deep learning model.

[0025] Output layer: Receives signals from the last hidden layer and outputs the model's predictions. The number of nodes in the output layer is usually determined by the nature of the task. For example, a binary classification problem usually has only one node, while a multi-classification problem has multiple nodes.

[0026] Stacked identity embedding formation: In this stage, the system obtains identity embeddings from different inputs and stacks them. Specifically, it first obtains fused embeddings from multiple input sources (such as text prompts and images). Then, these embeddings are concatenated along the length dimension to form a stacked identity embedding. This stacked embedding matrix essentially contains rich information about all identity features in the input. This processing method not only ensures the integrity of all input identities, but also their unified representation makes it possible to adaptively merge them in the next step.

[0027] Adaptively Incorporating Identity Content: In this step, the stacked identity embeddings are processed using all criss-cross attention layers to adaptively incorporate identity content from the diffusion model. In particular, the criss-cross attention layers implement global context modeling by computing the relevance weights of each embedding with all other embeddings. Each embedding then updates their representations based on these weights. This process effectively describes the complex interactions of identity content and helps to gain deep feature understanding.

[0028] The stacked fused embedding vectors are processed using a cross-attention layer, including: using the cross-self-attention mechanism to calculate the attention scores between the modalities, dynamically adjusting the weight distribution between different modal features, and strengthening the correlation between the stacked identity embeddings.

[0029] In this embodiment, the deep features are subjected to a multi-stage adjustment process, specifically: in the multi-stage fine-tuning stage, the goal is to achieve the best balance between editability and identity fidelity. To achieve this goal, the training of this stage involves the alternating use of real data and synthetic data at each stage. In the early stages, a large amount of data is mainly used for pre-training, and then higher-quality aesthetic data is used for review and fine-tuning in the later stages. This multi-stage training strategy can improve the text consistency and visual quality of the generated results, while avoiding falling into potential overfitting too early.

[0030] The multi-stage adjustment of deep features includes: the initial stage, using a wide range of data sets to cover identity pictures, generate diverse samples, and improve the model's adaptability to changes through random data augmentation; the training stage, introducing domain-specific data sets for learning details and special scenarios, while adjusting optimizer parameters.

[0031] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating consistent identity images based on a diffusion model, characterized in that: include: A multimodal large language model is used to generate optimal paired data, where the paired data includes text data and corresponding image data; a text encoder is used to encode the text data to obtain a text embedding vector; an image encoder is used to process the image data to obtain an image embedding vector; the text embedding vector and the image embedding vector are input into the MLP module to be fused to obtain a fused embedding vector; all the fused embedding vectors are stacked, and the stacked fused embedding vectors are processed using a cross-attention layer to obtain deep features; Multi-stage adjustments are performed on deep features to obtain consistent identity images.

2. According to the method for generating a consistent identity image based on a diffusion model according to claim 1, it is characterized in that: Using a multimodal large language model to generate optimal paired data includes: obtaining a reference image, inputting the reference image into the large language model, and generating text information; injecting expression and posture information into the text information to obtain multimodal text information; using a text generation model to process the multimodal text information to obtain a synthetic image; preprocessing the synthetic image, and associating the text information with the preprocessed synthetic image to obtain the optimal paired data.

3. The method for generating a consistent identity image based on a diffusion model according to claim 2, characterized in that: The use of a text generation model to process multimodal text information includes: the text generation model includes an input layer, a hidden layer and an output layer; the multimodal text information is input into the input layer for encoding processing to obtain a feature map; the feature map is input into the hidden layer to extract hidden features; the hidden features are input into the output layer, and after being restored by a decoder, a synthetic image is obtained.

4. The method for generating a consistent identity image based on a diffusion model according to claim 1, characterized in that: The text encoder is used to encode text data, including: using CLIP-T as the text encoder, adding position embedding to the text encoder to maintain word order information, and using layer normalization and residual connection to ensure the stability of training; in the last layer of the encoder, the self-attention mechanism is used to represent the features, and the representation results are used as embedding vectors.

5. The method for generating a consistent identity image based on a diffusion model according to claim 1, characterized in that: The image encoder is used to process the image data including: using CLIP-V as a feature extractor, multi-level features from underlying texture to high-level semantics, and compressing the extracted features into a 1D embedding vector through a global average pooling layer.

6. The method for generating a consistent identity image based on a diffusion model according to claim 1, characterized in that: The MLP module fuses the embedded vectors, including a multi-layer perceptron network including multiple hidden layers and a ReLU activation function; the multi-layer perceptron network is used to fuse the features to obtain a fused embedded vector.

7. The method for generating a consistent identity image based on a diffusion model according to claim 1, characterized in that: Stacking all the fused embedding vectors includes: concatenating all the fused embedding vectors along the length dimension to form a stacked identity embedding.

8. The method for generating a consistent identity image based on a diffusion model according to claim 1, characterized in that: The stacked fused embedding vectors are processed using a cross-attention layer, including: using the cross-self-attention mechanism to calculate the attention scores between the modalities, dynamically adjusting the weight distribution between different modal features, and strengthening the correlation between the stacked identity embeddings.

9. The method for generating a consistent identity image based on a diffusion model according to claim 1, characterized in that: The multi-stage adjustment of deep features includes: the initial stage, using a wide range of data sets to cover identity pictures, generate diverse samples, and improve the model's adaptability to changes through random data augmentation; the training stage, introducing domain-specific data sets for learning details and special scenarios, while adjusting optimizer parameters.

Images