Personalized face image generation method based on pre-training model LoRA weight generation

Abstract

The invention discloses a personalized face image generation method based on pre-training model LoRA weight generation, which comprises the following steps of: firstly, acquiring a training sample set according to actual requirements, constructing and training a LoRA weight auto-encoder and a training diffusion model, then, forming a LoRA weight generation model by a decoder and a diffusion model in the trained LoRA weight auto-encoder, when a personalized face image needs to be generated, generating a LoRA weight vector by adopting the LoRA weight generation model, extracting a low-rank decomposition matrix from the LoRA weight vector, calculating to obtain a weight updating matrix, and generating a personalized face image; and updating the weight matrix of the pre-trained text graph model and generating a personalized face image according to a text prompt. According to the method, the LoRA weight generation model is constructed in advance, and the LoRA weight is directly generated in the personalized face image generation process, so that efficient personalized image generation without fine adjustment is realized, and the generation speed and the image quality are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image generation, and more specifically, relates to a personalized face image generation method based on pre-trained model LoRA weight generation. Background Art

[0002] The goal of personalized text-generated images is to generate high-fidelity personalized images based on specific identity information provided by the user. Existing personalized image generation methods are mainly divided into two types. The first method is to generate personalized images through model fine-tuning, which requires additional training on specific reference images to ensure that the generated images can accurately reflect the identity characteristics specified by the user. Generally, such methods are based on large-scale text-to-image diffusion models and achieve high identity fidelity image generation through fine-tuning at the inference stage. The second method is to add additional trainable branches to the existing model to achieve personalized image generation. This method directly introduces identity information at the inference stage through a pre-trained auxiliary branch without additional fine-tuning. By leveraging this auxiliary branch, the model is able to generate personalized images containing specific identity features at inference time while maintaining high generation efficiency.

[0003] However, there are some problems with existing personalized image generation methods:

[0004] First, existing personalized image generation methods usually rely on model fine-tuning, such as LoRA (Low-Rank Adaptation). This method requires additional training in the inference phase, which leads to the following problems: (1) This fine-tuning process is usually time-consuming, which may take 10 to 30 minutes, making it difficult to meet the needs of real-time generation of personalized images. (2) Fine-tuning methods are highly dependent on large-scale datasets and are not effective when processing a small number or a single reference image. (3) Since fine-tuning involves retraining the model, the generation ability of the original model may be impaired during the generation process, affecting the final image quality.

[0005] In addition, existing methods achieve personalized image generation by adding trainable branches, but usually require freezing the weights of the original model, which increases the inference cost and leads to the following problems: (1) After freezing the model weights, the training cost of the auxiliary branches is high, which may cause additional computational overhead. (2) Adding trainable branches during the inference phase can easily introduce additional complexity, resulting in a decrease in the identity fidelity of the original generated model. (3) Existing methods find it difficult to achieve efficient personalization while retaining the generation capabilities of the original model. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a personalized face image generation method based on the pre-trained model LoRA weight generation. By pre-constructing the LoRA weight generation model, the LoRA weights are directly generated in the personalized face image generation process, thereby achieving efficient personalized image generation without fine-tuning, effectively improving the generation speed and image quality.

[0007] In order to achieve the above-mentioned object of the invention, the personalized face image generation method based on the pre-trained model LoRA weight generation of the present invention includes the following methods:

[0008] S1: Obtain several face reference images M according to actual needs ref , each face reference image M ref Input the pre-trained Wensheng graph model, use the LoRA weight algorithm to obtain the low-rank decomposition matrices A and B corresponding to its weight update matrix ΔW, and then generate the LoRA weight vector V using the following method LoRA :

[0009] For the two low-rank decomposition matrices A and B of the weight update matrix ΔW, the low-rank decomposition matrix A is flattened into a one-dimensional vector V by matrix flipping A , flatten the low-rank decomposition matrix B into a one-dimensional vector V B , then vector V A and vector V B Splice to get the LoRA weight vector V LoRA ;

[0010] Each face reference image M ref And the corresponding LoRA weight vector V LoRA Construct a training sample, thereby obtaining a training sample set;

[0011] S2: Construct a LoRA weight autoencoder, including an encoder and a decoder, where:

[0012] The encoder is used to input the LoRA weight vector V LoRA Encode and get the LoRA embedding feature F LoRA And send it to the decoder;

[0013] The decoder is used to embed the LoRA feature F LoRA Decode and reconstruct the LoRA weight vector

[0014]

[0015] Each LoRA weight vector V in the training sample set LoRA The LoRA weight autoencoder is trained as a training sample to obtain a trained LoRA weight autoencoder;

[0016] S3: Construct a diffusion model whose input is random noise and face reference image M ref , the output is to generate LoRA embedding features

[0017] For each training sample in the training sample set obtained in step S1, the LoRA weight vector V LoRA Input the decoder trained in step S2 to obtain the corresponding LoRA embedding feature F LoRA , embed LoRA into feature F LoRA After the noise processing, the noise embedding feature is obtained Then the noise is embedded into the feature as random noise and the corresponding face reference image M ref The diffusion model is input together, the diffusion model is trained, and a trained diffusion model is obtained;

[0018] S4: The decoder trained in step S102 and the diffusion model trained in step S103 are used together to construct a LoRA weight generation model, where:

[0019] The diffusion model is used to input the face reference image M ref Diffusion processing with random noise is performed to generate LoRA embedding feature F LoRA And send it to the decoder;

[0020] The decoder is used to input the LoRA weight vector V LoRA Encode and reconstruct the LoRA weight vector

[0021] S5: When a personalized face image needs to be generated, the face reference image and random noise are input into the LoRA weight generation model to obtain the reconstructed LoRA weight vector The LoRA weight vector will be reconstructed According to the inverse process of the LoRA weight vector generation in step S1, the reconstructed low-rank decomposition matrix is ​​obtained Reconstruct the low-rank decomposition matrix according to Calculate the weight update matrix The weight matrix of the pre-trained text-based graph model is updated to obtain an updated weight matrix W=W+ΔW; the text prompt is input into the text-based graph model after the weight matrix is ​​updated to generate a corresponding personalized face image.

[0022] The invention discloses a personalized face image generation method based on LoRA weight generation of a pre-trained model. First, a training sample set is obtained according to actual needs, each training sample includes a face reference image and a corresponding LoRA weight vector, a LoRA weight autoencoder and a training diffusion model are constructed and trained, and then the decoder and the diffusion model in the trained LoRA weight autoencoder are used to form a LoRA weight generation model. When it is necessary to generate a personalized face image, the LoRA weight generation model is used to generate a LoRA weight vector, a low-rank decomposition matrix is ​​extracted from it, and a weight update matrix is ​​calculated to obtain the weight matrix of the pre-trained text graph model, and a personalized face image is generated according to a text prompt.

[0023] The present invention has the following beneficial effects:

[0024] 1) The present invention realizes personalized image generation by directly generating personalized LoRA weights without additional model fine-tuning. That is, after pre-building the LoRA weight generation model, the reference image is used to directly predict the LoRA weights in the inference stage, and it is integrated into the existing Wensheng graph model, thereby realizing efficient and accurate personalized image generation, effectively solving the efficiency and fidelity problems in the prior art, and greatly improving the practical application effect of personalized image generation.

[0025] 2) In the diffusion model training, the present invention also introduces a face reference image enhancement processing module, which enhances the face reference image by combining facial features and overall image features, significantly improving the accuracy of identity information extraction and ensuring the accurate reconstruction of LoRA weights. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of a specific implementation method of the personalized face image generation method based on the pre-trained model LoRA weight generation of the present invention;

[0027] Figure 2 It is a structural diagram of the LoRA weighted autoencoder in the present invention;

[0028] Figure 3 is a structural diagram of the face reference image enhancement processing module in this embodiment;

[0029] Figure 4 It is a schematic diagram of generating a personalized face image in the present invention. DETAILED DESCRIPTION

[0030] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0031] Example

[0032] Figure 1 This is a flowchart of a specific implementation method of the method for generating personalized face images based on LoRA weight generation using a pre-trained model of the present invention. Figure 1 As shown, the specific steps of the method for generating personalized face images based on LoRA weight generation of the pre-trained model of the present invention include:

[0033] S101: Obtain training sample set:

[0034] LoRA weight is an efficient parameter fine-tuning technology, which achieves rapid customization of LLM (Large Language Model) by adding a small number of trainable parameters to the pre-trained model while keeping the original model parameters unchanged. The core of LoRA is that it decomposes the weight update matrix ΔW into two smaller matrices A and B, thereby approximating fully supervised fine-tuning with fewer parameters. In order to realize the automatic generation of LoRA weights in the present invention, it is necessary to train the constructed LoRA weight autoencoder and diffusion model, so it is necessary to obtain a training sample set, and the specific method is:

[0035] Get several face reference images M according to actual needs ref , each face reference image M ref Input the pre-trained Wensheng graph model, use the LoRA weight algorithm to obtain the low-rank decomposition matrices A and B corresponding to its weight update matrix ΔW, and then generate the LoRA weight vector V using the following method LoRA :

[0036] For the two low-rank decomposition matrices A and B of the weight update matrix ΔW, the low-rank decomposition matrix A is flattened into a one-dimensional vector V by matrix flipping A , flatten the low-rank decomposition matrix B into a one-dimensional vector V B , then vector V A and vector V B Splice to get the LoRA weight vector V LoRA .

[0037] Each face reference image M ref And the corresponding LoRA weight vector V LoRA Constitute a training sample, thereby obtaining a training sample set.

[0038] S102: Build and train LoRA weight autoencoder:

[0039] In order to realize the automatic generation of LoRA weights in the present invention, it is first necessary to construct and train the LoRA weight autoencoder. Figure 2It is the structural diagram of the LoRA weighted autoencoder in the present invention. Figure 2 As shown, the LoRA weight autoencoder in the present invention includes an encoder and a decoder, wherein:

[0040] The encoder is used to input the LoRA weight vector V LoRA Encode and get the LoRA embedding feature F LoRA And sent to the decoder. In this embodiment, the encoder uses a one-dimensional convolution layer.

[0041] The decoder is used to embed the LoRA feature F LoRA Decode and reconstruct the LoRA weight vector In this embodiment, the decoder uses a cascaded attention layer and a multi-layer perceptron to restore the original structure and information characteristics of the LoRA weight vector, where the attention layer is used to use the attention mechanism to embed the LoRA feature F LoRA Processing is performed and the obtained features are sent to a multi-layer perceptron; the multi-layer perceptron is used to generate and reconstruct the LoRA weight vector based on the received features

[0042] Each LoRA weight vector V in the training sample set LoRA The LoRA weight autoencoder is trained as a training sample to obtain a trained LoRA weight autoencoder. In this embodiment, the conventional mean square error loss function or cross entropy loss is used when training the LoRA weight autoencoder.

[0043] S103: Build and train a diffusion model:

[0044] Construct a diffusion model whose input is random noise and face reference image M ref , the output is to generate LoRA embedding features The face reference image is integrated into the adaptive layer normalization (AdaLN) mechanism in the diffusion model to guide the generation of LoRA embedding features. In this embodiment, the diffusion model adopts the DiT (Diffusion Transformer) model, which combines the Transformer structure and the diffusion model. It not only inherits the excellent expansion characteristics of the Transformer model, but also improves the performance by replacing the U-Net backbone.

[0045] For each training sample in the training sample set obtained in step S101, the LoRA weight vector V LoRA Input the decoder trained in step S102 to obtain the corresponding LoRA embedding feature F LoRA , embed LoRA into feature F LoRA After the noise processing, the noise embedding feature is obtained Then the noise is embedded into the feature as random noise and the corresponding face reference image M ref The diffusion model is input together, and the diffusion model is trained to obtain a trained diffusion model. The loss function of the diffusion model training can be set according to actual needs. In this embodiment, a mean square error loss function is used.

[0046] In order to improve the training effect of the diffusion model, in this embodiment, a face reference image enhancement processing module is constructed to participate in the training of the diffusion model, that is, the face reference image enhancement processing module is used to process the face reference image M in the training of the diffusion model. ref After enhancement, the data are input into the diffusion model. Figure 3 FIG. 1 is a structural diagram of the face reference image enhancement processing module in this embodiment. Figure 3 As shown, the face reference image enhancement processing module in this embodiment includes a face encoder, a face multilayer perceptron (MLP), an image encoder, an image multilayer perceptron, a feature splicing module, a gate network (Gate Network) and a feature fusion module, wherein:

[0047] The facial encoder is used to obtain a face reference image M ref Extract facial embedding features F face And sent to the feature splicing module and the facial multi-layer perceptron. In this embodiment, the facial encoder adopts the insightface encoder, which can obtain facial details related to identity recognition by extracting facial embedded features.

[0048] Facial multilayer perceptron is used to embed features F from the face face Extract facial features from And sent to the feature fusion module.

[0049] The image encoder is used to obtain the face reference image M ref Extract image embedding features F image And sent to the feature splicing module and the image multi-layer perceptron. In this embodiment, the image encoder uses the image encoder in the CLIP (Contrastive Language-Image Pre-training) model, and the global appearance features of the face reference image can be obtained by extracting the image embedding features.

[0050] Image multilayer perceptron is used to embed features F from the face image Extract image features And sent to the feature fusion module.

[0051] The feature splicing module is used to embed the face into the feature F face and image embedding feature Fimage Perform splicing to obtain the splicing feature F con And input into the gating network.

[0052] The gating network is used to con Calculate the facial score S face and image score S image , and then sent to the feature fusion module.

[0053] The feature fusion module is used to calculate the facial score S face and image score S image For facial features and image features Fusion is performed to obtain the enhanced face reference image

[0054] In practical applications, the face reference image enhancement processing module and the diffusion model can be trained together. In order to improve the training efficiency, some modules can adopt pre-trained models and then freeze their parameters. For example, in this embodiment, the parameters of the facial encoder and the image encoder are set to the pre-trained parameters and frozen, and the parameters of other modules are trained together with the diffusion model.

[0055] S104: Construct LoRA weight generation model:

[0056] The decoder trained in step S102 and the diffusion model trained in step S103 are used together to construct a LoRA weight generation model, where:

[0057] The diffusion model is used to input the face reference image M ref Diffusion processing with random noise is performed to generate LoRA embedding feature F LoRA and sent to the decoder.

[0058] The decoder is used to input the LoRA weight vector V LoRA Encode and reconstruct the LoRA weight vector

[0059] S105: Personalized face image generation:

[0060] Figure 4 Schematic diagram of the generation of personalized face images in the present invention. Figure 4 As shown in the figure, when a personalized face image needs to be generated, the face reference image and random noise are input into the LoRA weight generation model to obtain the reconstructed LoRA weight vector The LoRA weight vector will be reconstructed According to the inverse process of the LoRA weight vector generation in step S101, the reconstructed low-rank decomposition matrix is ​​obtained Reconstruct the low-rank decomposition matrix according to Calculate the weight update matrix The weight matrix of the pre-trained text-based graph model is updated to obtain an updated weight matrix W=W+ΔW. The text prompt is input into the text-based graph model after the weight matrix is ​​updated to generate a corresponding personalized face image.

[0061] In order to better illustrate the technical effect of the present invention, the present invention is experimentally verified using specific examples. In this embodiment, a test set containing 25 identities is collected, including 10 men and 15 women. The images of each identity do not appear in the training set, in order to evaluate the generalization ability of the model. In order to ensure the comprehensiveness of the experiment, this embodiment prepares 30 different text prompts for each identity, which cover a variety of scene descriptions from simple to complex. For each text prompt, 4 images are generated, which are used together to evaluate the stability and generation quality of the model.

[0062] In this embodiment, four prior arts are selected as comparison methods, including:

[0063] LoRA-DreamBooth: By combining LoRA’s low-rank fine-tuning technique with DreamBooth’s personalized image generation method, LoRA-DreamBooth can efficiently fine-tune large models and generate customized images with limited resources. For details, please refer to the paper “Ruiz N., Li Y., Jampani V., Pritch Y., Rubinstein M. and Aberman K. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242, 2022.”

[0064] Textual Inversion: Generates personalized images by embedding new text words into the latent space of the pre-trained model. It is suitable for personalized style transfer and text-to-image generation. For details, see the paper "Gal R, Alaluf Y, Atzmon Y, et al. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion[J]. 2022."

[0065] InsantID: By introducing a lightweight adaptation module combined with a pre-trained diffusion model, a single facial image can be used to generate a personalized image, keeping the identity features consistent, and achieving high-fidelity image generation without a lot of fine-tuning. For details, see the paper "Instantid: Zero-shot identity-preserving generation in seconds"

[0066] PhotoMaker: By combining textual cues and latent space encoding, the image encoder and decoder operate in the latent space to generate high-quality, semantically consistent images. For details, please refer to the paper "Li Z, Cao M, Wang X, et al. PhotoMaker: Customizing Realistic Human Photos via Stacked IDEmbedding[J]. IEEE, 2023. DOI: 10.1109 / CVPR52733.2024.00825."

[0067] When evaluating the performance of generating personalized face images, this embodiment uses the following indicators:

[0068] 1) DINO: used to evaluate the similarity between the generated face image and the face reference image in the high-level feature space to measure the identity fidelity.

[0069] 2) CLIP-I: Evaluates the consistency of visual features between the generated face image and the reference identity, which is also an important indicator of identity fidelity.

[0070] 3) CLIP-T: used to evaluate the match between generated face images and text prompts to measure text-image consistency.

[0071] 4) Face Sim: It is used to evaluate the similarity between the generated face image and the face reference image. In this embodiment, the two images are firstly passed through the insightface encoder to obtain face embedding features, and then the similarity of the face embedding features is calculated to obtain face sim.

[0072] 5) Image Reward: Provides a more comprehensive evaluation of text-image consistency.

[0073] 6) FID: Evaluates the realism and diversity of images by comparing the distribution differences between generated images and real images. Lower FID values ​​indicate higher image quality.

[0074] 7) Speed: This is the personalized generation speed, which is used to measure the time cost of the generation process. For methods that do not require fine-tuning, only the inference time is recorded; for methods that require fine-tuning, both the training and inference time are recorded.

[0075] Table 1 is a statistical table of evaluation indicators of the present invention on all test sets in this embodiment.

[0076]

[0077] Table 1

[0078] As shown in Table 1, the present invention has significant advantages in generating high-quality, high-identity personalized images, and is also much more efficient than the prior art.

[0079] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

Claims

1. A personalized face image generation method based on LoRA weight generation, characterized in that: The following steps are involved: S1: Obtain several face reference images M according to actual needs ref , each face reference image M ref Input the pre-trained Wensheng graph model, use the LoRA weight algorithm to obtain the low-rank decomposition matrices A and B corresponding to its weight update matrix ΔW, and then generate the LoRA weight vector V using the following method LoRA : For the two low-rank decomposition matrices A and B of the weight update matrix ΔW, the low-rank decomposition matrix A is flattened into a one-dimensional vector V by matrix flipping A , flatten the low-rank decomposition matrix B into a one-dimensional vector V B , then vector V A and vector V B Splice to get the LoRA weight vector V LoRA ; Each face reference image M ref And the corresponding LoRA weight vector V LoRA Construct a training sample, thereby obtaining a training sample set; S2: Construct a LoRA weight autoencoder, including an encoder and a decoder, where: The encoder is used to input the LoRA weight vector V LoRA Encode and get the LoRA embedding feature F LoRA And send it to the decoder; The decoder is used to embed the LoRA feature F LoRA Decode and reconstruct the LoRA weight vector Each LoRA weight vector V in the training sample set LoRA The LoRA weight autoencoder is trained as a training sample to obtain a trained LoRA weight autoencoder; S3: Construct a diffusion model whose input is random noise and face reference image M ref , the output is to generate LoRA embedding features For each training sample in the training sample set obtained in step S1, the LoRA weight vector V LoRA Input the decoder trained in step S2 to obtain the corresponding LoRA embedding feature F LoRA , embed LoRA into feature F LoRA After the noise processing, the noise embedding feature is obtained Then the noise is embedded into the feature as random noise and the corresponding face reference image M ref The diffusion model is input together, the diffusion model is trained, and a trained diffusion model is obtained; S4: The decoder trained in step S102 and the diffusion model trained in step S103 are used together to construct a LoRA weight generation model, where: The diffusion model is used to input the face reference image M ref Diffusion processing with random noise is performed to generate LoRA embedding feature F LoRA And send it to the decoder; The decoder is used to input the LoRA weight vector V LoRA Encode and reconstruct the LoRA weight vector S5: When a personalized face image needs to be generated, the face reference image and random noise are input into the LoRA weight generation model to obtain the reconstructed LoRA weight vector The LoRA weight vector will be reconstructed According to the inverse process of the LoRA weight vector generation in step S1, the reconstructed low-rank decomposition matrix is ​​obtained Reconstruct the low-rank decomposition matrix according to Calculate the weight update matrix The weight matrix of the pre-trained text-based graph model is updated to obtain an updated weight matrix W=W+ΔW; the text prompt is input into the text-based graph model after the weight matrix is ​​updated to generate a corresponding personalized face image.

2. The method for generating a personalized face image according to claim 1, characterized in that: In step S1, the encoder uses a one-dimensional convolutional layer.

3. The method for generating a personalized face image according to claim 1, characterized in that: In step S1, the decoder uses an attention layer and a multi-layer perceptron, wherein the attention layer is used to use the attention mechanism to embed the LoRA feature F LoRA Processing is performed and the obtained features are sent to a multi-layer perceptron; the multi-layer perceptron is used to generate and reconstruct the LoRA weight vector based on the received features 4. The personalized face image generation method according to claim 1, characterized in that: In the training of the diffusion model in step S3, the face reference image enhancement processing module is used to process the face reference image M ref After enhancement processing, the image is input into the diffusion model. The face reference image enhancement processing module includes a face encoder, a face multi-layer perceptron, an image encoder, an image multi-layer perceptron, a feature splicing module, a gating network and a feature fusion module, wherein: The facial encoder is used to obtain a face reference image M ref Extract facial embedding features F face And sent to the feature splicing module and the facial multi-layer perceptron; Facial multilayer perceptron is used to embed features F from the face face Extract facial features from And sent to the feature fusion module; The image encoder is used to obtain the face reference image M ref Extract image embedding features F image And sent to the feature splicing module and the image multi-layer perceptron; Image multilayer perceptron is used to embed features F from the face image Extract image features And sent to the feature fusion module; The feature splicing module is used to embed the face into the feature F face and image embedding feature F image Perform splicing to obtain the splicing feature F con And input into the gating network; The gating network is used to con Calculate the facial score S face and image score S image , and then sent to the feature fusion module; The feature fusion module is used to calculate the facial score S face and image score S image For facial features and image features Fusion is performed to obtain the enhanced face reference image 5. The method for generating a personalized face image according to claim 4, characterized in that: The face reference image enhancement processing module is trained together with the diffusion model, wherein the parameters of the facial encoder and the image encoder are set to the pre-trained parameters and frozen, and the other modules are initialized to the pre-trained parameters and trained together with the diffusion model.

6. The method for generating a personalized face image according to claim 4, characterized in that: The face encoder adopts insightface encoder.

7. The method for generating a personalized face image according to claim 4, characterized in that: The image encoder adopts the image encoder in the CLIP model.

Images